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x INTRODUCTION 

The passage of the Comprehensive Employment and Training Act of 1973 marked a 
major effort to establish a decentralized approach to ,the delivery of employment 
and training services. Under CETA, primary responsibility for planning, designing 
and operating employment and training programs was shifted from the federal to the 
state and municipal levels. Since the needs of local labor markets differ sub- 
stantially- across regions of the nation, it was reasoned thaf local policy and 
planning officials could, best identify the needs of their communities and most 
effectively tailor service strategies designed to address those needs. 

As part of this shift in Authority to the local level, prime sponsors are 
responsible for assessing the Effectiveness of thetr efforts by examining the 
extent to which CETA participatioVttas^nhanced the^ longer-term income and employ- 
ability status of former enrollees. Designing and Implementing Local Follow-Up 
System* has been developed to help prime sponsors meet this responsibility and 
consists of two parts. 



Part I: Minimum Recommended Prime Sponsor Follow-Up System contains : 

* ' < 

o A series of recommendations designed to facilitate the implementation 
of a* minimum follow-up capability; 

o A recommended participant follow-up questionnaire; and » 

o A set of sample data tables which can be used to facilitate the production 
of prime sponsor follow-up reports . 

Part II; Conducting Outcomes Evaluation on the tocal Level provides a more 

in-depth treatment of the issues covered in Part I and contains: N # ^ 

o An overview of program evaluation in the employment and training system; 

o A discussion of the necessary types and sources of local follow-up data; 



o A discussion of the actual operation of a follow-up system, including 
the design of survey instruments, the selection of an organizational 
approach, and the location, contact and interview of former CETA 
participants; and ^ v ^ 

o A review of issues/related to analyzing local follow-up data, including 
1 ' atf illustration of alternative analytical approaches. 
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AN OVERVIEW OF PROGRAM EVALUATION IN THE EMPLOYMENT AND TRAINING SYSTEM 

J '• ' • • 

I. THE/ ROLE AND TYPES OF PROGRAM EVALUATION 

The enactment of the Comprehensive Employment and Training Act of 1973 repre- 
* 

sented a major departure from the centralized and categorical nature of decision- 

1 

making and program structure that had previously , existed. Und^r CETA, basic 
decision-making authority was shifted from the U.S. Department of Labor to eligible 
units of 'state and local government known as CETA prime sponsors. This shift gave 
prime sponsors the responsibility for effectively responding to the employment and 
training needs of their local jurisdictions through the planning, administration 
and evaluation of their own employment and training programs. 

* To successfully execute this responsibility, prime sponsors must have a well 
developed set of " management capabilities, including that of program evaluation. 
Once the prime sponsor has identified the employment ^nd training needs of its area 
and has developed a services mix to meet. -specific local objectives, program evalua- 
tion is needed to determine how well programs work, and which ones work best, for 
whom, and ,at what costs. 

The assessment of program performance can be viewed from sever 
perspectives. While the evaluation literature contains references to,, and def 

tions of, many types of evaluation, the following four classes appear most 
2 

frequently: ' 

o Process Evaluation 

This class of evaluation focuses on program operations . It 
seeks to determine if program activities; e.g., assessment, 
training services, or job development, are being provided 
according to what was planned,. if the planned number and 
' types of participants are being served, and if the planned 
number and types of terminations are being achieved. 
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o Outcomes or fmpact'Evaluatiott ' 

t 

In contrast to process evaluation, this type focuses on program 
outputs and determines the degree to wh^ch the longer-terra goals 
of • a program, such as enhanced employment stability or gains in 
earned income, have been achieved* . 

o Comprehensive Evaluation k 

A comprehensive 'evaluation emerges m when -the results of process . 
and outqomes evaluation are % combined. This type of evaluation 
gives a more complete appraisal of program performance and can. 
provide substantial insight into reasons- for program success 
and failure, and #ays in which performance can be improved.,. 

o" Strategic Evaluation »- 

* 

This class of evaluation attempts to determine which programs 
work best by comparing the effectiveness of alternative 
» strategies designed to achieve similar objectives. ^ . 

While eaci), of these types -of evaluation has a role to play within the employ- 
ment and training system, policy^makers, administrators, and planners at each level 
•of government will need different types of evaluative information. For instance, - 

national officials can use- process evaluations to assess the flow of participants . 

* .3 
through prime sponsors ,/ and to make' inter-sponsor comparisons' of performance. In 

contrast, prime sponsors can rely on process evaluations to assess. JChe movement of 
participants through the local delivery system , its individual programs , and , when 
applicable , contractors . Thus, while the basic issues that determine ^accountability 
are the same across levels of government, the specific questions and information 
needs vary Distinguishing among these questions and needs is critical for deter- 
mining the types and nature of evaluation most appropriate on the national, state, 
and local levels. , * > * 



II. PROGRAM EVALUATION AT THE NATIONAL, STATE, AND LOCAL LEVELS * • 

On the national level, evaluations of the CETA program are necessary to enable 
the Department of Labor to fulfill several key management and policy-making respon- 
sibilities. As stated in the 1978 CETA Amendments: 
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"The Secretary shall provide for the continuing evaluation of all 
' programs, activities, and research a^d demonstration projects 
conducted pursuant to this Act, including their cost-effectiveness 
, in achieving the purposes of this Act, their impact on communities 

and participants, their implications for related programs, the* < 
1 > extent to which they meet the needs of persons by age, sex, race, s 
and national origin, and the adequacy of the mechanism for the 
t delivery of services.''^ ' 

/ . 

y To meet these responsibilities, evaluative information on the impact of the total CETA * 

system and its major subparts and programs is needed to determine whether established 
goals and objectives have been achieved. Nationally based evaluations are also 
necessary to* assess how well, and to what extent, the CETA system is serving those 
subgroups of the population identified as most in need of employment and training 
services. 

. National evaluation needs are generally met through the Continuous Longitudinal y 

Manpower Survey (CLMS) and the Department of Labor's formal quarterly reporting 
5 

system. While the CLMS uses individual participant data collected frota a representa- 

r 

tive sample of prime sponsors to generate estimates of program impact, the D.O.L. 
reporting system relies upon standardized prime sponsor quarterly reports to assess ~ 
the ongoing performance of local programs. Using these quarterly reports, the 
Department of Labor 'can .determine if local program services are provided in a manner 
consistent with that planned. 
* ' Although national evaluations may provide important information on the effec- 

tiveness ^of the CETA system, they are o* limited use to local administrators and ^ s 
planners due to the diverse nature of prime sponsors. Given the highly aggregate 
nature of national evaluations, they cannot disentangle important differences in * 
local program design and organization, or determine their influence on local pro- * « 
gram performance. Additionally, the categorizations of programs in nationally 
based evaluations may not closely match specific programs that prime sponsors 
operate on the lo.cal JLevel, A classroom training program in one sponsor may pro- 
vide long-Perm occupational training, while in another, it may consist of short- « 
terjn training combined with job search assistance services. 

O ' -3- . ' 
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Because prime sponsors hive the' responsibility for planning, administering, 
and evaluating employment and training programs within their jurisdictions, they 
\ • require evaluative .information that accurately reflects the nature and effective- 
ness of their own individual programs . As such, prime sponsors can use local 
evaluation to achieve operational control over programs, to determine the effec- 
tiveness of existing programs in achieving locally established goals and objec- 
tives, and to make deoisions*' among alternative uses of program res6urces. Within 
this context, local program evaluations should be designed to address the following 
questions: 

1. To what extent have the goals and objectives of the local 
delivery system been achieved? ■ 

2. Have the observed levels of performance been uniformly 

. achieved across individual programs and population subgroups? 
f • - 

3. How can the performance of certain programs be sustained and 
that of others be improved? 

By structuring local evaluation systems around these management questions, 
prime sponsors will be better able to manage, their local employment and training 

' delivery syst^ns, and to make those adjustments necessary to improve program 

performance. 1 * ■ 

♦ 

III, THE 1978 CETA AMENDMENTS AND LOCAL FOLLOW-UP EVALUATION 

Among the types of evaluation potentially useful to prime sponsors is outcomes 

evaluation. Outcomes evaluation is needed to assess the ability of the local 

delivery system to produce a desirable set of participant outputs- To achieve this, 

1 however, prime sponsors must develop and implement local follow-up systems to track 

the post-program labor force experiences of CETA terminees. 

In October 1978,. the Congress reauthorized the QETA legislation for four more 

years, making substantive change^ in the goals of the CETA program, its structure, 

and the administrative guidelines governing- program operations and prime sponsor 

7 

reporting responsibilities. From a local evaluative perspective, «,» ra!avdnt change 
was the incorporation of specific language iwt training and employment opportunities 

ERIC « . • 



provided to participants ",\ ♦result in an increase in their earned income". Similarly, 

services provided under Title II of CETA were "...to ease barriers to labor force^ 

- 9 % 
participatidn encountered by economically disadvantaged persons/ 1 Given that measur- 

ing the performance of local programs in both producing gains in earned income and 
easing barriers ,to labor force 'participation requires information about the post- 
program labor force and income status pf terminees, these changes' in the CETA legisla- 
tion clearly suggest the need for local participant follow-up surveys. ■ " * 

While the 1978 Amendments did not instruct prime sponsors to develop particular 
types of local follow-up systems, they did clarify the need for conducting foljow-uo 
surveys of termiuees. Section 127 (d)(4) of the legislation requires that prime 

sponsors report to the Secretary o£ Labor. "the types of outcomes that participants 

. , $0" * 

experience after*the program." Moreover, the current CETA .regulations explain 

thisnrequiremSht in Section 676.22 (d)(2) as follows: 

"(d) Each recipient shall establish and use procedures for the 

systematic assessment on a quarterly basis of program per- ' 
formance in relation to the goals contained in its grant. 
Recipients shall: 

♦ ..(2) Establish and use procedures for collecting performance 

information (including information on the status of ^ * . 

individuals subsequent to entering unsubsidized employ- 
ment) and for Assessing such information in terms of the I 
gpals in its grant... "I- 1 * I * 

These requirements are amplified in both the Forms Preparation Handbook (FPH) 

and a recent D.O.L. publication entitled, CETA: Management Information System Program , 

■ 12 

Functional Management Information System . While th§ language l^n the FPH does little' 
more than ifcform prime sponsors to use their discretion in deciding how to meet ttje 
follow-up requirement, the more recent publication clearly indicates the minimum types 
of follow-up data required to adequately assess longer-term program performance. 
Therefore, the main i3sue for prime sponsors is how to establish an effective fpXlow- 
up capability that will provide Useful information on program performance and support* 
nianagement decision-making. The primary purpose of this technical assistance guide § 

• i ' ■ ' 

is to address this issue. * , 
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DESIGNING A LOCAL FOLLOW-UP DATA BA: 



ERIC 



1 



I. INTRODUCTION , 

r 

The primary purpose of lo<fal follow-up is to provide those data necessary 

to measure a,nd analyze the longer-term "outputs of the local employment and training 

f V • 

system* When' designing a follow-up System, prime sponsors must first select the 

' . v 

types, nature, &nd sources of data to be' included in their follow-up data b^ses. 

Choices among tfie available alternatives will have a direct" bearing on the outcomes 

data available for analysis, the comprehensiveness and timeliness of the follow-up 

data, and their usefulness for local decision-making. 

'Jhe purpose>of this chapter is to present and discuss key issues involved in 

the design of a local follow-up data base. The discussion is divided into the 

following fout areas: % 

o ( Selecting the types and' natufe of data to be included in. a 
local follow-up system; 4 1 

o Selecting the types of tetaninee^s and the scope of program 
cbverage that 0 will f orm^ the basis of the follow-up effort; 

0 

o Determining the appropriate length of the follow-up period 
and .the frequency at which follow-up data will be collected; 
and • e 

o Choosing the. appropriate .sources of data, e.g., institutional 
sources or direct participant contacts, for usd as the primary 
means of gathering *follow-up data. * • 



a 



II, SELECTING THE TYPES AND NATURE OF- DATA f OR A LOCAL FOLLOW-UP DATA BA^€ 

For prince sponsors, outcomes evaluation is a management tool which provides 
specific information on the relative 'longer-term ^success of alternative program ^ 
strategies, which ones Worked best and for whom/ and the factors responsible for 
producing variations in 'program performance.* .With the information derived from 
outcomes evaluation, prime sponsors should be in a better position to make those 



a * 

adjustments necessary to Ifoprpve the performance offcheir delivery systems. 



-8- ' 



? 



As'was discussed in Chapter One, this view of outcomes evaluation can be 
expressed in 'three management questions: 

1. To what extent teve the goals and objectives of the local 
delivery system. been achieved? 

* 2. Have the observed levels 'of ^success been uniformly achieved 
across individual programs and 'population §ubgroups? 

3\ How can the performance of certain prograjns be sustained and ^ 
that of others improved? 

To address the first question, prime- sponsors must select those d^ta that be^t 

reflect the goals^and objectives of their local delivery systems. Since the primary 

goal, of CETA programs is to enhance the income* and employment status of ^program par- 

* ft 

ticipants, prime sponsors should focus their data collection efforts on alternative 
economic measures of local performance, such as those ^relating to the labor force 
status, employability , and earned income of terminees. As shown in Table 2-1, 
several types of labor market * related data are available for potential use by prime 

# * A 

/ 

sponsors, including the earned income of CETA terminees : As a measure of, program - 

performance, earned income is the most direct and 'inclusive indicator of program 

effectiveness in that it reflects one explicit goal of the CETA* program and is a 

* 1 

composite indicator of the total post-program labor for v ce experiences of terminees. 

There are Several reasons why prime sponsors should gather other types of labor 
jnarket related data in addition to participants' post-program earned income. The 

' , -4 

first is that a given level of earned income can be produced through various combina 
tions of participants' average hourly wages, weeks of employment, and total hours 
worked. If, for instance, post-program hourly wages and hours worked are the stanfe 
across participants, but weeks of employment differ, variations in earned income 
will be observed. Similarly, different levels of either or both hourly wages and 
hours will 'produce different levels of earned income for a given period of- employ- 
ment. Since there may be differences in the way local programs are designed to 
impact post-program earned income 1 , distinguishing among these income components is 



f 



\- Table 2-1: Potential Types of Outcomes Data Upon Which to 
. -focusf A Local Follow-Up, Pat a Base 



r 



Economic Measures 

Annual * EafSSjags * 

Hourly Wage Rates 
* 

Hours Worked Per Week 

i 

Weeks Employment 
Weeks of Unemployment 



Labor For.ce Status At 'Time of Follow-Up 
Interview ' 

' Occupational Mobility * 

Job Retentidh ' 

- iPublic Assistance -Status •- * * % 

4 ■ * 

Unemployment Insurance Status 1 



) 



Non-Economic Measures 

"School Enrollment Status' 

Participation in Another Training Program 



Enlistment in the Armed Forces 
Self -Conf id enc e * 



Job Satisfaction 



V 



-10- 



4) 

RIC 



18 



t • 2 

Important when- assessing program effectiveness. For example, in an occupational 
training program, services are likely designed to Impact participants 1 ' earned incomes 
by enabling them to secure higher paying jobs than they would have been able to with- 
out the program.. In contrast, an adult work experience program is probably geared to- 
impact its participants' ^earned income by improving their level of employment rather 
than their hourly wages. Complete reliance on earned income as an indicator of 
effectiveness may, therefore, not fully reflect the actual mix or objectives of local, 
programs, and could serve to obscure well informed judgements of local performance. _ 

Other reasons for broadening the scope of local follow-up data collection in- 
clude the likely neJ for both additional post-program performance measures and an • 
P 

understanding of participants' labor force behavior. In the first case, the 
' diversity of program objectives inherent in a local delivery system may very well 
require prime" sponsors to use several different performance measures when assessing 
prsgram effectiveness. These measures can include post-program job retention, .occupa- 
tional mobility, and t?he likelihood, of employment, all of which may^ reflect the., objec- 
tives of specific programs or the particular concerns of the prime sponsor and^its^ 

program operators. , , • ' , _ 

N The "main reason for using follow-up data to gain insights into^participants 1 
-post-program tabor 'force .behavior status is the need for feedback on how. to improve 
the performance of local, programs . Although judgements of program effectiveness^ can, 
be made with the use of specific performance measures, information on the events 
surrounding each measure can be quite useful for .identifying where programs are most 
- as well as lease effective. For example, a low level of long-term program effective- 
ness can be due to a rapid withdrawal of participants from the labor force. If this 
is the case, information as to why participants choose not to participate in the labor 
force will be -useful for identifying areas of program design that may Heed improvement. 
Issues related to participants' job search skills and self-confidence, as wftll as their 
day care and, transportation assistance needs, can be identified and linked back to the 

* » 

program to improve its design. 
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In addition to economic measures of performance, there are also non-economic 
aspects of p6st-program performance that prime sponsors should consider ' when design- 
ing their local follow-up data bases. *Table 2-1 lists several potential non- 
economic measures* 'For example, outcomes measures such as the level of participants u 
self-confidence and job satisfaction may be among those that prime sponsor progr^s 
are designed to impact. Moreover, participants 1 post-program school or training 
enrollment and military status are important type^f (positive) outcomes which many 
programs, particularly those serving youth and younger adults, may be designed to 
affect. In these instances, assessments of program effectiveness 'require the use of 

non-economic as well as economic measures of performance. 

\ 

While program effectiveness can be estimated through economic and non-economic . 

1 * 

performance measures, ddta on participants' views of the programs in which they ^ % 

participated represent additional information that can provide valuable insights '-into 

local performance. The participants 1 reasons for program enrollment, ratings of jro- 

grams, and views on the major strengths and' limitations of programs can be used • 

together with more objective performance data to guage why certain programs succeeded 

• < <* 

and others failed. 

' O^pe the outcomes data necessary for local follow-up purposes are identified, 
prime sponsors must determine the types of information needed to address the ,seco'nd 
major question concerning the uniformity of prog ram performance. This part of the 
evaluation process provides prime sponsors with the basis for making judgements of 
the relative effectiveness of their program strategies and the extent to which the 
local delivery system benefits .certain target groups more or less than others. The 
information required to successfully address this issue includes two major types. 
The first is data on the actual programs in which individuals participated, while 
the second includes selected participant characteristics and should be designed to 
reflect the mix of target groups identified for • services by the prime sponsor. The 
Department of Labor has already established some population characteristics as 



A - 

significant segments, including participants' race, age, and sex. Prime sponsors 
"should also consider using other characteristic data, such as participants levels 
of education, public assistance status, and prior work histories. 

While domparisSns program performance identify the strengths and weaknesses 
of the .local delivery system, - .these findings alone will not clarify whether 
differences in performance are due to the programs themselves, the characteristics 
of participants served in the programs, or the relative effects of individual program 
.strategies on particular population subgroups. To address these issues, prime 
sponsors must determine how the interaction of participants with programs results in . 
a particular' level of . performance Jo accomplish this objective, detailed data must 
be available-, as noted, on participants' characteristics and the nature and levels 
of their program participation. . ' • 

, 'Based on the above discussion, -a'lodal follow-up data base can be viewed as 
consisting of the. following five primary types of data: ' » 

o Selected personal, social, and economic characteristics of 

participants. • » ■ • 

o The type' and level of services received by participants. 

o The post-prograxa income, employment, and labor force 

experiences, of participants* t i 

o The non-economic aspects of program perf ormance^including ' ^ . 
areas such as school enrollment status and j^si^s faction, 

o Participants' view of the strengths' and palenesses of the 

programs in which they participated. 1 

Once prime sponsors select the types of data to include in a local follow-ujp 
data base, they should turn their attention to key decisions' regarding the nature 
of those data to be collected. These include choices between gross and net measures, 
and 'point-in-time and continuous^ erf ormance measures. These two issues are (addressed 
separately below. 
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A. Gross and Net Performance Measures ' 

<Jr*oss measures of program performance reflect the absolute or* total output 
of a program over the follow-up period. In the case of -earned income, the gross 
effect of a program is measured by participants' total earnings during the post- 
program period* Similarly,* other gross output measures includeji^e total number 

i 

of weeks of post-program employment experienced by terminees, their average hourly 
wage rapes, and the number and percent employed^ at a particular point in time. In 
contrast, net 'measures ^of performance are int^ndeji to represent the independent 
contribution of "program participation to the post-program 4 income and employment 
experiences of former participants. Since many, if not tnosi, program terminees 
would have obtained some employment and earnings in the post-program period, net 
outcomes are designed to reflect the differences between the actual experiences of 
trerminees and those that would have occurred without the program. In the case of 
earned income ^ the net program effect is expressed by the change in gross earned 
income attributable to participation £n a pro'gram. 

- There are two princ^le advantages associated with relying upon gross output 
measures to assess program performance. ,The first is that gross measures are 
relatively easy to Construct and do not require very sophisticated evaluatiye 
techniques. Second, if highly coi&related with long-ter& gains in employment and 
earnings, e.g. tVo years, -gross performance measures can prbvide a stiraightforwatvJ 

approach for determining whether a program's ^outputs are consistent with its 

4 * 
established objectives*. * * 

i 

Despitq^heir simplicity, gross performance measures have several conceptual 

< 

shortcomings. First, while may be possible to identify certain gross measures that 

are correlated with long-term improvements in participants 1 employment and earnings 

status, it will still not be fully clear whether or not and to what^ extent the 

observed outcome would have occurred without the program. A second shortcoming of 

gross measures is that they m^ create incentives for prime sponsors t</ serve<the 

* 

less disadvantaged members of their eligible populations. Since gross measures, 
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such as weeks of employmen^r hourly wage rates, are likely influence^ by factors 
such as the age, sex, and.educatij*6 of participants, there may be an incentive to 
serve those individuals who would probably h'ave relatively positi^ labor market 
experiences even in the^ absence of the program. Such groups could include white 
males and those with substantial prior work experience. _ ^ % 

In contrast, net program measures address these 'shortcomings . This approach 
provides the basis for determining the independent contribution of program participa- 
tion to the post-program labor force experiences of CETA terminees as well as aruin- 
• centive to serve the more disadvantaged segments of the eligible p&pulation. Since 
the less disadvantaged usually perform well in the labor market, it is difficult for 
local programs to generate 'impressive gains in their levels of earnings and employ- 
ment/ As a result*, prime sponsors would have an incentive to recruit and enroll 
th$se individuals with^less rewarding pre-program 'employment and earnings experiences 

There are certain practical issued that prime sponsors should consider when 

using net perfonpance measures. Most important is that precise measurements of net „ 

program outcomes require the availability of either a control or comparison group 

to provide a basis for determining what the employment and earnings experiences of 

* 5 
participants would have been in the absence of program participation. The observed 

differences between the actual post-program labor market experiences of participants 
and those of t^p control or comparison group represent the net effect of program 
participation. From a practical perspective, however, the use of either control 
or comparison groups for local follow-up purposes may be beyond the resources, 
expertise, and program flexibility available to prime sponsors. 

Without control or comparison groups, net measures of performance can be 
derived by comparing the pre- and post-program labor market experiences of the 
participants themselves. These before and after comparisons have been used in 
previous evaluations of employment and training programs although they do possess 
several important shortcomings. First, any observed pre- to post-program improve- 
ments in the employment and earnings of participants may simply reflect the passage 
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of time, including increased maturity of younger participants, rather than the 

effects ox the program alone. Second, differences between the pre- and post-pro- 

gram period may result partially from changes in local economic conditions rather 

than from program participation. ' Third, since some participants enter the CETA 

program after reaching a low point in their labor force and income status , part of 

the observed pre->post difference may reflect a movement back toward their more * 

permanent career status. In these cases, before and after comparisons will over- 

8 

state the effect of a program. / 
The above discussion points out a number of important trade-offs in choosing - 
between gross and nit measures ofperformance. Conceptually , net outcomes measures 
are most desirable since they can isolate 'the unique contribution of program par- 
ticipatiod to terminees' employment and earnings experiences. Moreover, using net 
output measures may create incentives for prime sponsors and their' program operators 
to serve the more disadvantaged members of the eligible population. However, . 
practical considerations suggest that the resources, expertise, and program flex-, 
ibility needed to construct precise estimates of net program outcomes may be too 
burdensome for many prime sponsors. v 

In light of these issues, prime sponsors may want to consider the use of gross , 
performance measures. In this case, it is important, at a minimum, to identify those 

gross measures' that are highly correlated with long-term gains in employment and 
9 

earnings. Although this precaution will not eliminate the .potential tendency to 
serve the less disadvantaged, it will provide a basis for, determining whether or not 
lqcal programs are performing in a manner consistent wiih their long-term goals and 
objectives. In this way, the gross performance measures of alternative programs can 
be compared in order to assess their relative effectiveness in producing desirable ( 
participant outputs. 
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B. Continuous and Point- in-Time Performance Measures * 

* — - ^ 

Prime sponsors must also decide whether to focus upon point-in-time <6r con- 
tinuous measures of program performance. Point-in-time measures , such as partici- * 
pants f labor force status or hourly wage rates' at a particular juncture during the 
follow-up period, are simple to use. Since such measures do not require a detailed 
account, of participants' post-program labor market activity, data collection efforts 
can be held to a minimum and will not necessitate the construction of more complex 
performance measures, such as earned income* Also, the use of point-in-time 
measures can be quite helpful for assessing the performance of prime sponsor delivery 
systems. Eor* instance, by comparing the labor force status and hourly wage rates of 
terminees at different points during the follow-up period, prime sponsors can begin 
to judge the effectiveness of their delivery systems in impacting the "longer-term 
employment and earnings o5 CEIA terminees. Moreover, comparisons of these measures 
across alternative program strategies will allow local planners and administrators 
to gain substantial insights into the relative effectiveness of those, programs. 

Despite their advantages, point-in-time measures provide only a limited picture 

of participants' post-program labor force activity. For example, some individuals, 
■ *• ^ 

unemployed at a^p^rticular point during the follow-up period may have been employed 

i 

for several, if not ail of the preceeding months since program termination. In such 
cases, evaluative judgements based solely on labot force status &ay understate the 
true level of .program effectiveness. The sam§ type of problem prevails with other 
point-in-time measures, such as hourly wage rat.es or hour^ "worked. 

In this regard, continuous performance measures , such as total weeks worked, 
average earnings, or hourly wages, provide a more comprehensive and accurate assess- 
ment of program effectiveness. Although such measures will require a detailed 
account of the post-program labor force activities of terminees, they will give local 
planners' and administrators more insight into how participants have allocated their 
time between employment and not*-employment. Table 2-2 provides a summary of potential 

types of outcomes measure^, by time dimension and type of effect to be measured. * 

-»< 
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Table 2-2: Potential Types of Outcomes Measures 



Type of Measure 



Gross Effect 



Net Effect 



Po int-in-Time Measures 



o -Employment status at time of 
follow-up contact 



<^ Iticrease^L likelihood of 
r ' employment at time, of 
follow-up contact 



o Hourly wage at time of follow- o Increases in hourly earn- 
up contact * ings at time of follow-up 
■ , , »' contact 

o Job retention status of initial o Increased likelihood of 
placements at time of foUow-up placements remaining with 

contact ' initial employers 



Continuous Measures 

J 



6 Total weeks worked during 
follow-up period > 



o Gains in number of weeks 
worked during follow-up 
peridd 

v v 



Total earned income during the o Gains\ in earnings during 



follow-up period 



follow-up period 



1 1 



9 
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III. SELECTING PARTICIPANXSUTO BE INCLUDED IN' THE LOCAL" FOLLOW-UP DATA BASE 

After selecting *hose data which should be included in a local follow-up base, 

prime sponsors need to identify those particip*ants "who will form the. basis of their 

follow-up *ef fort, When addressing this issue, prime spon*ors-will have to make ,the 

following key decisions: 

o Whether to group participants by their data of enrollment, or 
by date of termination; 

o ' Whajt types of terminees to follow-up; 

o What leve£*tfid scope of program operations to include in 
the foll^w-u^survey; and 

o Whether to employ sampling procedures. 

Each of these four issues is addressed separately below. 

'A. The Use of Enrollment and Termination Dates for Selecting Participants 

The first decision in selecting participants for follow-up is whether to group 
them Recording to the dates of their termination or the dates of program enrollment. 
If participants are classified according to their dates of enrollment, the follow-up 
survey will focus on individuals who were in the labor market during, a similar time 
period prior to the program . Since differences in pre-program labor marker *condi- 
tions can influence measurements of post-program per£dn&ance, using enrollment dates 
to select participants is one way to avoid biased judgements of program performance. 

In such. cases, however, the length of the follow-up period will vary system- 
atically with the duration of program participation. For example, assume that | 
follow-up data is collected eighteen months after enrollment for individuals served 
,in an .occupational training program and in a job search assistance program. Since 
job search assistance programs tend to be much shorter than occupational training 
programs, the length' of the post-program period' for both groups of participants 
will.be jquite different. Because differences in the duration of labor force par- 
ticipation will likely influence individuals 1 employment and earnings experiences, 
variations in the follow-up period alone can result in differences in the observed* 
levels pf success of the two programs. 

54 ..... . ... 



Another approach is Co group participants according to their dates of termina- 
tion* This assures that all participants will have been in the labor market for a 
uniform time period, und thus provides each program with an equal chance to work 
prior to any measurements of effectiveness- Since participants 1 post-program labor 
market activities form the -basis for measuring outcomes, reliance on the date of 
<ermination may be the most effective and reliable approach* for grouping program 
participants. 

B. Selecting the Types of Program Terminees 

The second issue requiring consideration is the selection of the types of 
terminees'for whom data will be collected. Choices must be made between focusing 
dkta collection efforts on all terminees, job placed terminees, other positive 
terminees, or those individuals with ''other 11 terminations. A key criterion for 
making this decision is the acquisition of data necessary for determining why 
certain programs wer.e more or less successful than others. - l 

Using this criterion suggests that data on the post-progSaULabbr market 

l 

experiences of program "failures" as well as "successes" must be available.. While 
this inclusive approach costs more than one focused upon a single groupjf 
terminees, it will enhance the utility of follow-up for decision-making. A local' 
follow-up system will therefore be of maximum use to prime sponsors if it is based 
upon all types of- terminees . 

♦ 

C. Selecting the Level and Scope o f Program Operations 

The level and scope of program operations for which post-program,data will be 
- made available will have an important bearing on the cost, required sample size, 
and usefulness of the local follow-up system for decision-making. Prime" sponsors 
must choose whether to f «cus their follow-up efforts on the entire Title IIBC - 
.delivery .system, its major program areas, such aa -occupational trainlng^and work 
experience, or on individual programs. ^ . 
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Although aggregate measures of performance will aid in judging the overall 
success of the local delivery system, they will be of limited use for identifying 
where changes are necessary ox; determining those adjustments needed to improve 
effectiveness- Since local employment and training -delivery systems are comprised 
of a varied of diverse programs often operated by several different service pro- 
viders, a useful follow-up data base is .one focused upon individual program 
activities - 

Beyond identifying the level of program operations for which follow-up data 
will be collect ed* prime sponsors must also decide the scope of programs to be in- 
eluded in their follow-up systems. While the most promising approach would be to 
include data on all programs in the local delivery system, this must be weighed 
against existing financial limitations and the objectives of each program. To the 
extent that the local delivery system is devoted primarily to the provision of par- 
ticular types of programs, such as occupational or on-the-job training, prime ^ • 
sponsors may wish to forego collecting data 'on other .less substantial program 
expenditures. • 

^tfdditional factors that should be considered are the objectives of individual 
local programs. Many local programs, such as basic education and pre-vocational 
training, are designed to prepare participants for entrance integer -programs 
geared specifically at enhancing their post-program employment- and earnings status. 
Although follow-up data can be used to measure the post rP tograft .labor market . 

experiences of individuals from these more "preparatory"^ of "feeder" programs,, their 
practical use for decision-making will be limited since the programs^ objectives are 



geared primarily at in-program gains in such ar6as as education and work attitudes. 

r 

D. The Use pf Sampling for Local F o11ow-Up Purposes^ , 
fdShe final consideration in selecting terminees f6r a local follow-up data base 
is^Sp^Sr not: and to what extent the prime sponso*r ( should rely upon sampling. 
The purpose of . sampling is to reduce the costs and scope of data collection while 
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maintaining the accuracy <Df the follow-up findings. Statistical results derived 
from a sample of participants should accurately reflect results which would be 
obtained by a study employing -the entire population of participants. Whether* or 
not sampling is possible depends' directly upon the number of participants a prime 
sponsor has enrolled in its programs ♦ 

Typically, small prime sponsors will not always have enough participants ' 
enrolled in their programs to.be able to. take advantage of Sampling procedures. 
When attempts are made to sample participants from an already small level of prq- 
gram enrollments, the accuracy 'of any follow-up findings will be tenuous at best. 
For larger prime sponsors, sampling is most desirable for those activities with 
•enough participants suCh- that -the resulting sample is representative of the total 
population. For instance, participants in occupational training programs may be 
sampled if the sample can be accurately stratified by certain participant 
cWa'ract.eristics of interest (e.g. age," race and sex), and is large enough to be 
representative of the entire* enrollee population. In such cases, prime sponsors 
can realizes substantial, swings relative to the costs of conducting follow-up for 
all participants'. A more detailed consideration of the'issUes involved in sampling 
^fs contained in Appendix S. « 

' iv. ' IDENTIFYING AM APPROPRIATE FOLLOW-UP PERIOD 

Since the primary purpose of a local f oj-ldw-up system; is' to help prime sponsors 

^make informed judgements of program effectiveness, local administrators and planners 
.will place, a premium on the timely availability of outcomes information. However, 
the need for current fpllow-up information must be balanced- against considerations 
regarding the accuracy of the follow-up findings and .their usefulness for portraying 
.the longer-term effectiveness of local programs. Therefore, tSe post-program period 

« must be long enough to reflect the permanent effects fhat programs have on partici- 
pants' employment' and earnings experience. ' Short . follow-up periods, -i.e. one to 

i three months, tend to reflect only the immediate topact of a program and provide 

■ C . ' ■ 
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little- assurance that the impact will continue over time* By extending the follow- 

up period to six, twelve, or eighteen months , prime sponsors can derive a more 

accurate basis for assessing the longer-term effectiveness of local programs. 

# 

Prime sponsors must identify a follow-up period t;hat can both meet their needs 

for timely information, and "'serve as the basis for making accurate judgements of 

post-program performance. Several existing studies show that post-program performance 

measures derived f rcp\six month follow-up information reliably predict gains 'in 

employment and earnings one year after program termination. While longer follow-up 

periods may be even 1 more reliable, the major advantage of a six month follow-up 

period is that it will generate timely information, some of which will cover the out-^ 
* . 13 " ' 

puts of current programs. 

V. SOURCES OF DATA FOR LOCAL FOLLOW-UP 

This section discusses various alternative data sources that prime sponsors 

* * 

tan use to meet their follow-up information needs. The discussion focuses "dn 'the 
, advantages and disadvantages of the most widely used data sources in light, of the 
five basic Criteria which are presented in Table 2-3. When reviewing possible data 
sources fot use in local outcomes evaluation, prime sponsors must; determine those 
criteria of greatest concern, and choose their data sources accordingly. A summary 
of the main sources for each type of data discussed previously in this chapter is 
provided in Table 2-4. 

A. Sources of Outcomes Data * v 

v One major type of data .which should be included in a prime sponsor follow-up V 
.data base is measures of program goals and objectives. These measures can be 
designed to reflect economic aspects of performance, such as earned income or total 
weeks worked, or non-economic program outcomes, such as^ job satisfaction, attendance 
in school, or enlistment in the Armed Forces. When selecting an appropriate data 
source to* meet these information needs,' prime sponsors can choose from four major 
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Table 2-3: Criteria' for Selecting Data Sources for Local 

Outcomes Evaluation 



I. Scope of Data Elements Available ■ v 

Data sources should be sufficient in scope to meet the follow-up informa- 
tion needs of prime sponsors. 

II. Quality of Data 

4 * 

Dat^ sources should be thoroughly reviewed for their accuracy, complete- t 
ness, and internal consistency. 

III. Repr esentat iveness ' 

Data must be'available for *all participant groups selected for the local 
follow-up data/base*. 




IV. Timelin€gs of Information Retrieval 



Data must be available in a tjjo^ly enough fashion to meet prime sponsor 
annual planning needs. 

< 

Cose 

Data sources should be reviewed and compared to determine that which is 
least costly,, given the information needs' identified by the prime sponsor. 



A 
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Table 2-4: Alternative Sources of Data for Local Follow-Up Systems 



Type of Data 
I. Post-Program Economic Data 



Sources 

o Social Security Administration 

Earnings Data 
o State Unemployment Wage Records 
o Employer Fpllow-Up Surveys 
o Participant* Follow-Up Surveys 



II, Post-Program Non-Economic Data, Including % # . 
Participants' Views of Programs 



o participant Follow-Up Surveys 



III, Demographic and Socio-Economic Information 



o Prime Sponsor MIS 

o Participant Follow-Up Surveys 



IV. Pre-Program Labor Market Data 



o Prime Sponsor MIS 

o Participant Follow-Up Surveys 

o Social Security Administration 

Earnings Data 
o State Unemployment Wage Records 



V.^ Program Activity Data 



o Prime Sponsor MIS 
o Prime , Sponsor CIS 
o Participant Follow-Up Surveys 



VI. Participant Location/Contact Information 



o Prime Sponsor MIS 
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sources: Social Security earnings data, Unemployment Insurance wage records data, 
employer surveys, and participant follow-rup surveys. The advantages and limitations } 
of each of these data sources are summarized in Table 2-5, and are discussed below . 
1. Social Security Earnings 'Data 

The\Social Security Administration (SSA) maintains a master 'file of yearly 

earnings for all employees nationwide in those jobs covered by Social Security. This 

file is arranged by Social Security number and contains selected demographic informa- 

tion, suc^ as sex and birthdate^ obtained when individuals request Social Security 

numbers. . Currently, this file is updated annually when employers file Copy A of their 

employer W-2 Wage and Tax Statement with SSA for those ^employees covered by Social 
14 

Security., - 

> 

Social Security earnings data possess many advantages as a, source of. inf orma- 
tion on participants' post-program employment and earnings experiences. SSA records 
are relatively free from any biases due to samp^ acfcritioti as coverage is nation- 
wide and includes approximately, 90 percent o^ all paid employment. Since SSA files 
* 

are maintained through employer tax returns, these records are also complete and 

accurate for all covered employees. Additionally, Social Security records are both 

longitudinal, giving those prime, sponsors interested in performing long-term evalua- 

tions the opportunity to readily do so, and relatively inexpensive, with the only 

costs involved being those of data processing and analysis.* 

This source of d*ta does, however, have several limitations. First, the 

measures of economic outcomes available from the data are limited to annual earnings, 

with no information on wage rates, weeks worked, or other aspects of post-program 
15 

performance. Similarly, information is not available on any non-economic measures 
- of performance. Second, Social* Security data are currently subject to at leasj a 
one and a half year delay in availability. For example, information on terminees' 
1980 earnings would not become available in the file until mid-1982. Third, about 
ten percent of all employees are not covered by Social Security, including federal 
and many state and local government employees, as well as employees of"«some non- 
profit organizations. For covered "employees, only earnings up to the maximum tax- 
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, Table 2-5: Strengths and Limitations of Alternative Data Sources 

for a Local Follow-Up Data Base 



r 



' Major Sources of Data 



Sources of Post-Program Data 



I. 



Social Security Earnings 
Data 



c 



II, 



Unemployment Insurance tfage 
Records Data (Stat^ Employ- 
ment Security Agency) 



III. Employer Follow-Up Surveys 



Advantages 



Available for most 
nationwide 
Accurate 
Complete 
Longitudinal * 
Relatively inexpensive 



Limitations 



employees o Measures of economic 
- outcomes limited to 

annual earnings 
o Data not available on 
non-economic outcomes 
o At l<east 1*5 year time 
lag in availability 
from year 'Vf reported* 
earnings 
o Ten percent of 

employees not covered 
o Repeated study proce- 
dures not possible 
o Data not available on 
par€icipants not 
employed 



Earnings redorded quarterly 
Only three month lag in 
availability from time of 
reported earnings 
Accurate 
Longitudinal . 
Relatively inexpensive 



o Relatively easy to contact 
former participants 

o Relatively high response rate 
possible 

o Accurate job and wage informa- 
tion 

o Can be used to supplement 
j other data sources 



Measures of economic 
outcomes limited to 
earnings 

Data not available on 
non- economic outcomes 
12 states do not main*- 
tain complete files of 
•U.I.* data 

Small firms and self- 
employed not covered 
Data not available on 
employees working out 
of state 

Data not available on 
participants not * 
employed 

Data available on job- 
placed terminees only 
i Data limited to first 
post-CETA job 
- o Difficult to identify 
appropriate respondent 
within establishment 
Data not available 
uon-economic outcomes^ 
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Table 2-5 (Cont.) 



Major Sources of Data 

IV. Participant Follow-Up 



. v Advantages 

Relatively short time lag 
Wide range' of data potentially 
available 

Flexible and subject to change 
Can elicit participant opinions 
and suggestions about programs 
Can Obtain information for all 
'types of term^nees 
Can be used to verify other 
data sources 



Limitations 

Relatively higher cost 
per participant contact 
Possible non-response 
biases 

Problems may be 
encountered in locating 
and contacting partici- 
pants 



Sources of Participant and 
Program Characteristics 

I. Prime Sponsor MIS 



Readily available 

Many data elements required 

by federal law 

Current 

Only readily available data 
source for some information 



o Not always complete. 

o .Full participant con- 
tact information not 
required by law 

o'Data in MIS not always 
consistent with local 
follow-up information 
need£ 



II . Prime Sponsor CTS 



o Readily available 
o Current 

o Concise picture of program 
activities 



o Detailed information 
on participants' 
experiences not avail- 
able 
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able for Social Security are recorded in the file. The fourth limitation is that 
no information is available for terminees who have not worked since^leaving CETA, 
or who have worked* so little that "their earnings are not recorded. As a result, 
follow-up findings obtained by, using SSA data could be biased toward those partici- 
pants with post-program earnings. 

A final concern is the way in which prime sponsors can access the data. Since' 
SSA records cannot be released to the public, the Social Security Administration 
must perform all statistical analyses* , The SSA provides a service which matches ^ 
the earnings data to identifying participant characteristics, and then performs 
requested statistical analyses* This procedure, is generally a less expensive means 
of acquiring follow-up data than survey techniques, but it substantially restricts 
the range of analyses which can be performed* 

Prime sponsors which wan^mpre information about using Social Security earn- 
ings data should contact the Office of Research and Statistics in the U.S. Social^ 
Security Administration. 

2. Unemployment Insurance Wage Data 

Most states, through their State Employment Security Agencies (SESA) , keep 
files of quarterly and yearly earnings for employees covered by Unemployment 
Insurance (U.I.). Like data available . frqm the Social Security Administration, ^ 
the U.I. wage records represent one potential institution source of follow-up 

data. , v 

* — - i 

U.I. wage data have advantages similar to those of Social Security earnings 
data. Information in U.I. wage records is complete, accurate; and longitudinal for 
those U.I. covered employees. who continue working in. the state. In contrast, how- 
ever, U.I. wage data are reported "quarterly , with earnings information generally 
available only three months after the end of the quarter, a much shorter time-lag 
' than that of SSA data. .This series' of quarterly earnings is a major feature of 
O.l/dfiETintha^.it present's a "clearer picture of an employee's labor market 
expediences than the yearly earnings available from Social Security data. 
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r U.I. data also have certain important limitations* Like Social Security earn- 
ings "data, U.I, wage data contain only earnings information and are not useful for 
examining other aspects of pbst-program -performance ♦ Additionally, U.I. wage data 
are not fully representative. Many employees of small firms, and those who are 
self-employed. are not covered by unemployment insurance. Additionally, data are 
not available for employees working outside of the state, or for terminees who have 
not worked since* they left CETA.' A final disadvantage of U.I. wage data is that- a 
dozen states, including some major industrial states, such as New York and Ohio, do 

not routinely record quarterly wage data for all covered employees in a master 
18 

file. Rather, these states acquire wage data for determining benefit rights on a 
request basis after the worker files a claim. 

• Actual data elements in a state's U.I. wage files vary across states, as will 
procedures to be followed to access the data. Prime sponsors interested in using 
U.I. wage. data for outcomes' evaluation should contact their State Employment Security 
Agency (SESA). . 
3. ' Employer Follow-Up Surveys 

In addition to the institutional data sources, outcomes information can also 
be obtained through the use of employer surveys. These surveys have several 
distinct advantages. Employers are generally easy to locate and contact', and most 
will take the time to answer the survey questions, resulting in a relatively high 
response rate. Also, employers can be relied upon to provide accurate Job descrip- 
tions, -wage rates, and time worked by their employees. 

. Despite these advantages,, however, employer surveys are quite limited in 
scope. Since the name of the first post-CETA employer is generally recorded only 
'-for participants who are job placed, it is not possible to obtain follow-up in- 
formation pn non-placement terminees/ Additionally, for those who were job placed, 
only the first post-CETA employees listed, limiting follow-up data to that job. 
Only when placements remain on their initial jobs 'for the entire- follow-up period 
can participants' entire post-program employment and earnings experiences be 
recorded. 
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4. Participant Follow-Up Surveys ^ 

Participant follow-up purveys represent the , most viable approach for collect- 
ing outcomes data available to prime sponsors. Most importantly., direct participant 
contacts afford the opportunity to collect the widest range of available information 
when compared to any of the three data sources already discussed. 

Several types of data can b.e collected through participant surveys, including 
those related to the post-program employment and earnings experience of terminees, 
information about non-economic program outcomes, such as school entollment and mili- 
tary status,. and respondents 1 views and suggestions about the programs in which they 
participated. In addition, participant follow-up surveys can be structured to 
provide timely information consistent with prime sponsor . annual planning needs,' and 
can be used to obtain information on all 'types ^of terminees, unlike SSA data, U.I. 
data, or employer surveys.' 

At the same time, problems do 'exist with relying on participant follow-up * 
surveys as a source of outcomes data. Fiprst,* there are various issues involved in 
settingnup. and operating ar survey, some of which will be discussed in Chapter Three. 

■r 

A major concern relates to locating and contacting terminees. Participants may 
have moved without leaving a forwarding address, may keep irregular hours which 
make them difficult to reach, or may simply choose not to participate in the sur- 
vey. This difficulty in locating and contacting terminees may introduce bias into 
the follow-up findings as the results may be based on disproportionately large and 
small shades of particular population subgroups. Another limitation of participant 
surveys is related, to the coSt of data collection. Operating participant surveys 
tends 'to cost more than using employer surveys or Social Security or Unemployment 
Insurance earnings data. 

Despite these limitations, however, the data potentially Available from 
participant surveys represen£ the most complete information on participants' post- 
program employment and earnings experiences, their views of the programs in which 
they participated, ' and non-economic aspects of performance. 

* -31- • 

, ^ 30 > 



I 

1 

In summary, selected economic outcomes measures can be obtained through 
several sources, including SSA and U.I. earnings files, employer follow-up surveys, 
and participant follow-up surveys. However, the first three of the sources provide 
information only on participants who*were employed after they left the CETA program; 
and even. in these cases, problems of 4ata coverage still exist. For prime sponsors 
to acquire adequate information on economic and non-economic? outcomes, as well as 
permittees 1 perceptions of all the programs in which they participated, participant 
follow-up surveys are the most viable alternative. 

B. Sources of Demographic ancf Socio- Economic Participant Characteristics , 

Preprogram Labor Market Data, and Program Participation and Contact Information 

Without sujveying CETA terminees, the information maintained by the prime 

sponsor in its Management Information System (MIS) and Client Tracking* fys tern (CTS) 

represents the only data Source available for participant and program characteristic 

information. The CTS presents a concise picture of the participant's CETA program- 

experience while the MIS provides a wider range of more detailed information. - These 

two syst ems should, for the most part, contain those data necessary to identify the 

characteristics of participants as well as the programs' in which they participated. 

Several types of contact information needed for locating and contacting 

terminees should also be available from the prime sponsor MIS, All MIS should coj 

tain participants f full names and addresses- Many systems also contain parti^l- 

pants 1 phone numbers and even "emergeficy" phone numbers of family member|^or friends • 

It should be noted that prime sponsor MIS are not .retired to record participants' 

phone numbers although they are critical pieces of contact information. Other types 

of MIS information useful for locating and contacting' former participants include 

the names and addresses of employers for those job placed, and schools or training 

programs in which the participant may have enrolled; upon termination from the CETA . 

.program. , 
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Although prime sponsor MIS fcnd CTS are a primary source of information on 
participants' program and pre-program experiences, they have limitations which may 
affect their potential use. Management inf ormation ^and client tracking systems 
exist to provide the data needed to fulfill federal 'reporting requirements* Such * 
information is often insufficient for many decisions which must be m&de at the 
local level. For example, federal regulations do not require that prime sponsors 
record the industries in which on-the-job training takes place* However, for 

4 

local planning efforts, this information can be valuable for determining whether 
the^uccess or failure of an OJT program is attributable, in part, to the industry 

m 

in which the participants are working* In addition,' such critical data items as 
the respondent's telephone number and alternate emergency numbers* of fa^l^f 
friends are also not required. Prime sponsors planning to use participant surveys 
to gather post-program information should aim to have these data included in their 
MIS as an important part. of their local foilow-up systems. 
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FOOTNOTES TO CHAPTER TWO 



The composite mature of earned income stems froto the fact that it is based 
upon the product of an individual'^ total weeks worked, average hours 
worked per week, and average hourly wage rates* As a measure of post- 
program performance, it therefore reflects the total labor force experience 
of former participants, as well as several objectives which a local 
delivery system may be designed to achieve. * 

For a thorough review of issues and methods related to analyzing the 
sources of income gains see: 

* « 

* 

Sum, Andrew, Katherine Mazzeo, Francis McLaughlin, and Jeffrey Zornitsky, 
Evaluating the Performance of Employment and Training Programs at the Local 
Level , Employment and Training Administration, U.S. D.O.L., Boston, 
Massachusetts, 1978, Volume Two, pp. 357-371. . 

While the 'conceptual reasoning behind this point may be clear, the useful- 
ness of the, approach will be critically dependent upon the types of 
statistical techniques ^selected for analysis purposes. For a'tftorough 
Review of the application of alternative statistical techniques see: 

; * 

Sum, Andrew, et al. > op * cit . , Volume Two, pp. 221-317. 

Thus far, a determination of Aiose gross performance measures that reliably 
predict long-term program effectiveness has not been satisfactorily 
resolved. Currently,^ only three published empirical research studies on 
this subject have been completed, each with somewhat differing results. 
See: f 

. » 

Borus, Michael, "Indicators of CETA Performance", Industrial and Labor 
Relations Review , Volume 32, No. 1, October, 1978, pp. 3-14. 

Gay, Robert, Validating Performance Indicators for Employment and Train- 
ing Programs , U.S. Department of Labor, Employment and Training Administra- 
tion, Office of Research and Development, Washington, D.C., April, 1978. 

Geraci, Vincent, and .Christopher King, Employment and Training (CETA) 
Program Performance: Long-Term Earnings Effects and Short-Term Indicators , 
University of Texas at Austin, Department of Economics, December, 1980. 

A control group is defined as a group of individuals who, when compared to 
those receiving services, are different only in that they did not receive 
program services. True control groups are established by randomly assigh- 
ing individuals to receive or not receive program services^ When random 
assignment is not possible, attempts can be made to match individuals 
according to those characteristics which likely influence program per- 
formance. This latter procedure results in a comparison group which, in 
contrast to a control group, represents an incomplete accounting of 
differences between individuals. For a good discussion of selecting con- 
trol and comparison groups see: 



Campbell, Donald, "Reforms as Experiments", American Psychologist , Vol. 
24, No. 4, 1969, pp. 409-429, 

Issac, Stephen, and William Michael, Handbook in Research and Evaluation , 
Edits Publishers, San Diego, California, 1971, 




Hardin, Einar, "On the Choice of Control Groups", in Borus, Michael 
(Ed*), Evaluating the Impact of Manpower Programs » D.C*. Heath and 
Company, Lexington, Massachusetts, 1972, pp. 41-58* 

Sax, Gilbert, Foundations and Educational Research , Prentice Hall, 
Englewood Cliffs, New Jersey, 1979. 

When using \ comparison group, the incomplete accounting of 
differences between individuals makes it necessary to use statistical 
controls in order to accurately identify the independent contribution 
of a' program to a given outcomes measure. • In most cases, evaluators* 
have utilized either multiple regression analysis or analysis of 
variance techniques to accomplish this. For a review of selected 
'Studies that have used and/or discussed the relative merits of alterna- 
tive comparison groups see* ( 

Bonis, Michael, John Brennan and Sidney Rosen, "A Benefit-Cost Analysis 
of the Neighborhood Youth Corps: The Out-of-School Program in Indiana M , 
Journal of Human Resources , Vol. 5, No. 2, Spring 1970, pp. 139-159. , 
(Uses eligible applicants -who did not participate in the program) ♦ 

Cooley, Thomas, Timothy McGu^re and Edward Prescott, The Impact of 
Manpower Training Programs on Earning: An Econometric Analysis , prepared 
for Offige of Policy, Evaluation and Research, U.S. Department of Labor, 
Washington, D.C., 1975. (Uses both eligible applicants who did not par-* 
ticipate in the program and a matched sample from Social Security files). 

Garey, Robert B. , The Earnings Effects of CETA Title I in South Carolina : 
An Evaluation Using Social Security Data , The University of South 
Carolina, Bureau pf Government Research and Service, Columbia, South 
Carolina, 1978. (Uses a matched sample from Social Security files). 

Gibbard, Harold, and Gerald Somers, "Government Retraining of the 
Unemployed in West Virginia", in Somers, Gerald (Ed.), Retraining the 
Unemployed , University of Wisconsin Press, Madison, Wisconsin, 1969, 
pp. 17-24* (Uses a matched sample of persons having active or inactive 
applications on file with the State Employment Service). 

Hardin, Einar, and Michael Borus, The Economic Benefits and Costs of 
Retraining , D.C. Heath and Company, Lexington, Massachusetts, 1971. 
(Uses non-trainees who applied and were eligible for the program but 
did not participate). 

Main, Earl, ^'Nationwide. Evaluation of M.D»T.A. Institutional Job Train- 
ing", Journal of Human Resources , Vol. 3, No. 2, Spring 1978, pp. 159- 
170. (Uses friends and relatives of" trainees who were unemployed at 
about the same time the participants entered training) . 

West;at, Inc., Continuous Longitudinal' Manpower Survey: The Impact of 
CETA on Participants Earnings » Working Papers One and Two , prepared for 
the Office of Program Evaluation, Employment and Training Administration, 
U.S. Department of Labor, Washington, D.C, 1980, (Uses matched sample 
from Social Security files and Current Population Survey). 
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For a review of several studies containing before and after- comparisons 
see: 

* 

Perry, Charles R. , Bernard E.\ Anderson, Richard L. Row^an, and Herbert R. 
Northrup, The Impact of Government Manpower Programs: In General » and 
On Minorities and Women , Industrial Research** Unit, The Wharton School, 
University of Pennsylvania, 1975. \ 

Additionally,* a critical review of before and after 'comparisons can be 
found in: / 

Sax, Gilbert, o£. cit. a 

One way to possibly address these problems is to employ a time-series 
model* The *key advantage of this approach is that it seeks to extend 
the pre-program period for a sufficient duration to account for the 
problems "of maturation and regression toward the mean. It does, how- 
ever,- still suffer from the problem of history. Moreover, the use of 
extended pre-program observations will not be fully possible for in- . » 
dividuals who either are new entrants to the labor market or* have not' 
participated in the labor force for ah extended period of time. In 
the first case, one could delete these individuals from estimates- of 
program effects - While a similar approach can be taken for the rer 
entrants, it may be desirable to, attempt to go farther^ack in time , 
to collect their work history data. 

See for instance: 

Geraci, Vincent, o£. 'cit . 

Pre-CETA labor market conditions can potentially influence the measure- 
ment of post-program performance in one or a combination of two ways. 
First, to the extent that local labor markets become loose, individuals 
equipped with marketable skills and experience may find themselves un- 
able to secure or maintain employment. In this case, the deterioration 
in their employment and income status would tend to be morl^ temp or afy 
in nature than a reflection of their permanent status in the labor 
market. Thus, measures oi; .program effectiveness could potentially be 
biased by the problem of "regression toward the mean" since these in- 
dividuals couHlfoe expected to improve their labor market status in the 
absence of program participation. Secondly, changes in local labor 
market conditions between the pre- and post-program period could either 
enhance or reduce an individual's labor market status in the absence 
of the program. In this case, the use of enrollment dates would serve 
to establish a consistent basis for judging the effectiveness of local 
programs. . 

While it is difficult to objectively classify prime sponsors by size, 
the key factor that shcfuld be used £o determine the appropriateness of 
sampling is whether oa: not a sample can be drawn to generate statistically 
reliable results. F<jr a thorough treatment of sampling techniques and 
procedures see: 

Cochran, William G«^ Sampling Techniques , John Wiley and Sons, Inc., 
-New York, Jfew York, 1963. 
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12. See: Seraci, Vincent, oj>. dt . ' , 

13. Even a six month follow-^p^perioS, however, cannot be fully relied upofl^to 
generate infonnatio^en all programs operating in the same year during 
which the follow-up survey is taking place. Assuming that follow-up begins 
in October as do new fiscal year programs, jtfie first follow-up of October 
termitiees could not beg^Ln until April, which 'is the beginning of the annual 
planning cycle^ Thus,' even wi,th a relatively short follow-up period such 

' as six months, prime sponsors: Will have to rely upon a mix of current as 
well as previous fiscal year data; 

-14. A copy oj^the W-2 wage and tax statement is contained in Appendix A. 

15. Prior to 1978, the Social Security Administration maintained a file of 
earnings by quarter as well as yearly earnings up to a maximum taxable 
level for a lr employees. In 1978, the current system of updating the 

file once< a year through employer W-2 wage and tax statements was initiated. 

• 

16. In early 1981, the maximum taxable infcome was increased to $29,700, which 
is likely sufficient to cover most CETA participants 1 post-program earn- 
ings. 

17. Prior to 1978, individuals were required to earn a certain amount in each 
quarter to be credited for the quarter* Currently, ar\ individual is given 
credit for one quarter fot each $£00 he/she earns. If an individual earns 
less than $300 in -a year, no annual earnings will be recorded for that 
year. 

* • 

18. As of the end of 1980, those states which do not maintain a complete file 

of U.I. wage data are: Hawaii, Massachusetts, Michigan, Minnesota, Nebraska, 
New Jersey, New York, Ohio*, Rhode Island, Utah, Vermont and Wisconsin. 
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IMPLEMENTING AND OPERATING LOCAL FOLLOW-UP SYSTEMS 




I. INTRODUCTION ', 

Prior' to implementing a locally-based follow-up -system, prime sponsors 
must make a series of decisions regarding the organizational and operational approach 
they will use. Since prime sponsors can .exercise considerable latitude in making 
these decisions, it is important that all options are explored fully ^and that both 
in-house and external resources are considered. 

This chapter is designed to help prime sponsors in the following major aspects - 
of their local follow-up systems: 

o The selection of a participant contact method; 

o The development of a follow-up questionnaire; 

o The choice of an organizational approach for con- ^ 
ducting follow-up surveys; 

o The use of participant contact and interview 
techniques; and { 

o The use of quality control and data processing 
. - procedures. 

Each of these topics is addressed separately below. 

II. SELECTING A PARTICIPANT CONTACT -METHOD 

There are three primary participant contact methods whi^h prime sponsors can 
use to administer their follow-up questionnaires: 1) telephone surveys; 2) personal 

» 

visit interviews; and 3) mail surveys. As shown in Table 3-1, each of these methods 
possesses several strengths and limitations that impact upon their response rates, 
data quality, and costs. When selecting a participant contact method, prime sponsors 
should consider these three factors since they will have a direct bearing on the . < 
usefulness of follow-up -information for decision-making. This section will provide 
a brief discussion of these factors as well as the results from an empirical study 
designed to determine the most effective and cost-efficient survey method for con- 
tacting CETA terminees. 



Table 3-1: Advantages and Disadvantages of 
Alternative Survey Methods 



Method 
Telephone: 



Personal Visit: 



Mail Surveys: 



Advantages 

Opportunity to directly 
question the participant 
Offers opportunity to ask 
respondent t;o clarify 
answers 

Relatively easy to manage 
and supervise; provides 
opportunity to control 
interviewer bias 
No travel required 



o Opportunity to directly 

question the participant 
o Offers opportunity to 

clarify answers 
o Interviewer is able to 

obtain visual clues* from 

respondent 
o Visual materials can be 

used for questions 
o Provides maximum* opportunity 

to influence response rates; 

Interviewer can travel to, 

search for participant 

o ' Wide geographic area can 
J>e covered economically 

o Greater privacy since 

questionnaire* is completed 
without an "interviewer 

o Questionnaire can be com- 
pleted at the leisure of 
the respondent 

o For those respondents who 
often may not be at home, 
mail contact may be more 
successful 



'.'0 



Disadvantages 

Length of interview 
must be limited 
Notability to obtain 
visual clues from 
respondents 
Visual materials caj 
be used for questions 
Limited ability to 
locate participant; 
not able to go to 
participant's residence 

Management of survey 
operations is" difficult 
Quality control and ' . 
verifications are more 
complicated 
Time-consuming 
travel is required 



Limited ability to influence 

response rates 

Limited ability to control 

data quality 

No opportunity to ask 

respondents to clarify 

answers 

Questionnaire may be 
ignored by those who have 
reading problems 
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The response rate of- a contact method is a measure of the number of terminees 
who complete a questionnaire relative to the tot;al number of terminees the prime 
sponsor attempted to survey. Generally, one would expect both telephone and 
personal visit surveys to yield higher Response fates than mail surveys. Since mail 
surveys rely solely on the respondent to complete and return the questionnaire, in- 
terviewers do not have the oppprtunity to contact terminees and directly solicit 
their participation in the survey. 

The overall quality of the data, including how completely and accurately the 
follow-up information has been recorded on the questionnaires, is also likely to be, 
higher for telephone and personal visit surveys. For both of these methods, inter- 
viewers who are familiar with the survey instrument are relied upon to ask all ques- 
tions, record responses and check f or -accuracy and completeness. In contrast, the 
respondent who completes a mail questionnaire at home is not familiar with it, and, 
therefore, may not understand what is being asked, or where to record the information 

Although both the quality of the data and the response rate are expected to 
be lower for mail than for telephone or personal visit surveys, mail surveys are 
generally thought to be a useful participant contact method because of their lower 
cost. Because all questionnaires are mailed to and completed by respondents them- 
selves, there are no costs associated with either administering the^questionnaire 
or traveling to and from the respondent's home. 

T 

In order to help prime sponsors make a selection fronT among these alternatives, 
the Employment and Training Administration, U.S. Department of Labor, recently ' 
supported a research project to determine the most reliable and cost effective 

t 

method of gathering post-program data using a participant questionnaire.. The pur- 
pose of the study was to compare and contrast each of the three, survey techniques 
with respect to their overall response rates, the consistency and completeness of 
the data collected, and the associated cost. 
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A* Findings From a Study of the Reliability and Cost-Effectiveness of Three 
Survey Methods 



The study included 784 former CETA participants who had terminated from five * 

prime sponsors located in two ETA regions. All participants^were economically dis- 

t - 
advantaged and terminated from programs funded under CETA Title IIBC (75 percent) 

and Title IID (25 percent)* These participants were randomly assigned to one of 
three groups corresponding to the three survey techniques .^This assignment deter^ 
mined whether the ensuing six mont^i *post-program follow-up interview attempt would 
be conducted iri person, over the telephone, or through the mail. Guidelines were 
provided to. project interviewers to insure that location and contact prqcedures were, 
uniformly pursued across the three alternative approaches* The results are briefly 
discussed below, while a detailed description of both the project and its findings 
is contained in Appendix B. - 
' f A comparison of the rates of response (actual survey completion) associated - 
with the three survey methodologies revealed fchat the highest response^rate was . \ 
achieved by the telephone survey (45 percent), followed closely by personal visit 

interviews (AO percent). The mail survey yielded only a 19' percent rate of 

1 

response after a two stage mailing. Analysis of the telephone and personal sur- 
vey methods indicated that, although there was a slight difference in the response 
rate, the actual rate of contact (including those who were contacted but did not 
respond to the 'interview) was identical (48 percent). This indicates that, com- ^ 
pared to the telephone survey sample, a large group of those ^pproached for 
personal Visit interviews refused to participate. Some of the refusals came 
during the initial phone contact as an attempt was made to schedule a personal 
interview. It appears that respondents prefer the anonymity of a telephdne or 
mail survey to the face-to-face contact of personal interviews. Also, many 
terminees failed to meet a pre-scheduled personal interview appointment. This 
underscores the importance of taking advantage of the initial contact success by 
attempting to conduct the interview immediately. 
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Another major focus of the research was to assess the "quality" of data N 
collected with respect to its completeness and internal consistency, including the 
appropriate use of skip patterns and instructions. Similar to the response rate 
findings, the telephone and* personal visit interview methods generated high levels * 
of data quality and consistency. In contrast, the information collected through 
, - • mail surveys yielded -less complete and less consistent data. These r^sjilts are 
consistent with expectations since telephone and personal visit interviewers 
received extensive training in the use of the survey instrument ,• whereas maij. sur- 
vey respondents were(viewing the questionnaire for the first time. (^^^ 

The^final major issue under examination was relative cost. The cost of a 
completed interview was calculated to include the time spent locating terminees, , 
the actual time spent administering interviews, and the -time spent checking- and , 
coding completed questionnaires. The least expensive approach was the telephone 

survey, with art average cost of $2.13 per completed interview. The personal visit 

* . ... 

. method was considerably more expensive, witfr *an average cost of $16.27 per Completed 

interview, clearly reflecting the considerable time and travel costs associated with 

■ this approach. The average cost of -an interview completed through.a mail survey was 

found -to be $6.59. This mall survey cost may seem high since there are no associated 

interview costs; however, the two mailings coupled with a low response rate resulted 

in a comparatively costly operation. ^ 

These findings demonstrate that the telephone survey technique most favorably 

• •* 

balances the major considerations of resppnse rates, data quality^and overall costs. 
Although personal interviews produced competitive response ratas and- high quality 
data, the associated high costs keep them from being a practical option for prime 
sponsor use. Similarly, mail surveys do'not emerge as .a viable local option because 
they tend to generate low response rates and comparatively low quality data. There- 
fore, prime sponsors should primarily consider telephone surveys when selecting a ^ 
j 



participant contact method. 
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III. DEVELOPING A FOLLOW-UP QUESTIONNAIRE 

The follow-up questionnaire is the primary source of information on the 
post-program labor market activities of CETA terminees. Because this information 
serves as the basis for evaluative judgements, the survey instrument must be 
constructed to maximize the accuracy and utility of the follow-up data base. This 
section on questionnaire development coversjthe following three areas: 

o The development^and wording of questions; 

o The advantages and limitations of^ several methods of 
collecting information on the participants' post- 
' program activities; and 

o' Issues in constructing the questionnaire. • 

On occasion, an issue in questionnaire 'development" is critically linked to 
the choice of a particular survey method. When considerations are notably different 
among telephone, personal visit and mail'prvey Instruments, the variations will be 
\i discussed. 

» * * 

i 

A. ^ Developing Questions % 
t 

Once the list of desired post-program data is determined, questions must be 
developed to elicit exactly this ^information from CETA terminees. Questions must 
be precisely focused and phrased so that the average respondent can. easily under- 
' stand what is teing a»ked. Careful consideration should also be given to the 
length of the questions. To keep the interview as short as possible, questions 
should be designed to cover all the required information without undue overlap or 
repetition. -This section discusses first the basic types of questions which can 
be used in developing « follow-up questionnaire, and then focuse^n issues in 
the wording of questions. 
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1. Types of Questions 
t 



There are three broacL categories of/question and answer formats: fixed % 
response, open-ended', and sortTngor "skip" questions. 



.Fixed response questions are those for which a series of possible answers « 
is listed below the question. By having the most common responses, written below 
the question, the interviewer can record the respondents 1 answers' very quickly 
and then proceed with the rest of the interview. This approach, however, is 
only possible when the f>rime sponsor knows what types of responses are likely. 

When administering the 'interview, the interviewer either reads the choices 
to the respondent,, or matches the respondent's answer with one of the listed 
responses* * 



"How did you find out about this job?" 






1 - CETA Agency 


5 


■ Friends or Relatives 


2 * Employment Office/Job Service 


6 


- Answered Newspaper Ad 


3 * Private Employment Agency 


7 


- Other 


A - Contacted Qnployer Directly {j 






\ 







3 



In this example, responses are not read aloud as they may influence the 
respondent's choice of an answer. There is a blank lirje next to "other" to be 
used for listing, responses other than those already identified. It is necessary 
to accurately record these "other" responses, since they represent an important 

t " 

additional source of information ind may be listed as additional choices in the 
future. 

Another type of fixed response question seeks to direct . and focus the 
response by providing the only possible choices to the respondent. 



r 
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Example 2 _ 

, t 

"All things considered, how would you rate the CETA^program?" 

1 « Excellent 2 - Good 3 = Fair 4 » Poor 

It should be noted that on a questionnaire used in a mail survey, there is 
less of a difference between types of fixed response questions because the 
respondent always reads the responses* On questions for which there may be more 

answers than those listed, an "other ' " chqice may be included 

with instructions to "please specify". 

Open-ended questions are those for which no. predetermined responses are 
listed on the questionnaire, or read aloud to the respondent. There are two 
varieties of open-ended questions. The first is the open-ended question for whiph 
the respondent provides a numeric amount. 

• ««- 

r 

Example 3 - - 

"What was your starting hourly wage rate?" 

or 

"How many hours do you usually work per week?" . , 

,In both of these cases, the answer itself is a numb,er and is a readily usable 
data element either ..as recorded, or after being grouped into discrete categories. 

*. The second type of Open-ended question involves a non-numeric response. The 
interviewer asks the respondent the question and then records the entire reply 
verbatim, as in Example 4. * 

Example 4 . . — 

• "How would you improve the CETA -program?" 



5 
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This type of question is more time-consuming than fixed response, or 
numeric open-ended questions, due to the potentially extensive nature of the 
responses. "However, it provides respondents with the greatest degree of lati- 
tude when responding to' a particular question* 

Certain survey'questions can be developed as either a fixed response or 

. aji opetv-ended question as shown in Example 5* * 
♦ 

Exaraole 5 " { ; . . 



A) 1 M How many hours do you usually* work per week?" 
l B) - "How many hours do you usually work per week?" 



V r- 0-15 . % *• % 4 = 36-45 

2 - 16-25 > . . 5 - 46-55 

. 3 « 26-35 ' i> / u. 6 = 56+ 



Determination of the most appropriate format depends both upon the. time con^ 
straints of the interview as will as the nature of the analysis'. to be performed 
with the data. In th^ above example, it is prbbably faster to record the number 
of hours instead of scanning the alternatives in Option B for the appropriate 
resppnse- Developing a* questionnaire also requites foresight "as to future uses 
, of tfte .dat^. Usually, questions which are not categorized on the survey instru- 
ment^ (^ti^ A ritzier "than .Option B above), allow the analyst more latitude and 
% ' «' ■ • ' . v./ . 1 * 

/* • * \ ir^ t - v '* \ h f °" 

\Cfl«xibliity ! in aggtegfcting* airi ^^aggregating the data for various analytical 



- 5 
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Aitfiird type of ^uerstioii* is t-h*Lsort#fe or "skifc^ question. These questions 
aire generally ufced to'obtiain pne small piece o^f i^ormacW to ; direct the inter- 
viWer 6p the riesct appropriate groufc Jpf, qitesj;ibns, v . , , . . ■ 



i < r » « . 

A'* \\ ' *• „ \ » , 6;'-*' ^ > 



1^ 



"Are you currently working 



1 - Yes 2 - No 



This question sorts out respondents who are working from those who are not y 

The interviewer can then, for instance, proceed to questions about the current job 

<# 

** 

for respondents who .answer "yes" arid to questions about job search activities for 
those who answer "no". Sorting questions used oh a mail questionnaire indicate to 
respondents the next set of applicable questions or instructions. Sorting ques- 
tions and their uses for the "skip pattern" of the questionnaire are discussed in 
greater detail in the third part of this section, Constructing the Questionnaire. 
- 2. Wording of Questions and Responses 

There are several considerations to keep in mind when writing the questions 

for a participant follow-up questionnaire.* Questions should be simple, clear /and 
— * 

easy^todhderstand, and: should avoid the use of technical language or other jargon 
They should have a defined scope and should not seem vague to the respondent. 
Questions 1 should also be objective and not make the respondent feel that there is 
a preferred answer* 

Some examples illustrating a few of these major concerns are shown below. 



A) 


"In the m 


weeks since 


you 


left the 


CETA 


program, how many 


weeks 


were you 


unemployed?" 












B). 


"In the 


weeks since 


you 


left the 


CETA 


program, how many 


weeks 




were you 


not working?" 













In Question A, the term "unemployed" is used. Most respondents will probably 
treat the question as asking them when thefy were "not working", but some respon- 
dents may understand it tc^mean/'not working and looking for work." Consequently, 
respondents may answer the question from two p^fir^ctives. Question B eliminates 
this possibility. 
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Example 8: Defined scope, not vague. 



A) "Why did you enroll in the *CETA program? 

B) "What was your main reason for enrolling in the CETA program? 



.9" 

9" 



Essentially, Questions A and B ask for the same information. However,* . * . 
Question B narrows the scope of the question to one primary reason which focuses 
the respondent's recollection and is easier for the interviewer to record than 
several contributing reasons. 

Example 9: .Simply phrased. 



A) "What were your starting and final hourly wage rates for this job?" 

B) "What was your starting hourly wage rate for this job?" 
"What, was your final hourly wage rate?" 



Questions which are short and straightforward are easier to follow than longer 
onesT" Tn some cases, as in Example 9, this may mean asking two simple questions . 
instead of one more complicated one. 

Example 10: Objective 



A) "You haven't been looking for work every week since you left your ^ 
last job, have you?" 

B) "Have you looked-for work every week since*you left your last job?" 



Questions should be phrased' as objectively as possible. While question A 

has an obvious slant, there t may be instances where the slant is more subtle. 

• Concerns about question wording may vary depending on the survey method. 

Questions need to be very short and very simple when read ^over* t he telephone. In 

personal interview, questions can be longer and more detailed as the respondent 

* 

c^n see the interviewer's facial expression and make eye contact during the inter- 
view. On a mail survey, the language used for questions must be extremely basic, 
although questions "may need to be longer in order ^to contain sufficient explanation 
or instructions to allow fbr a response in the absencVo^any assistance from an 



interviewer. 



i 
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Other concerns in structuring a mail questionnaire are question, format and 

t * 

the wording of response choices* In any survey instrument, questions should be 
easy to read and should be clearly distinguished from the other text. This j.s 
particularly tzrucial in the mail questionnaire. Questions must be clearly 
designated and techniques, such as capitalization or underlining of key points, 
used when appropriate and possible,. as in Example 11. v 



J 



Example 11 



A) "Whar was the best thing about the CETA program?" 

B) "What was the BEST thing about the CETA program?" 



Wording of response choices in fixed response questions is primarily a con- 

/ _ _ _____ 

cern in questionnaires sent through the mail. Whereas response choices in tele- 

phone and personal visit questionnaires can include technical" terms, or abbrevia- 
tions ^mkil questionnaire response choices must be as clear and easy to under- 
stand as Xhe questions themselves.^ _.For example, one response under the question 



7lt 



"How did yajuiiad out about the first job you 'held after leaving the CETA program? 
might be the listings maintained by the local State Employment Security Agency, 
often referred to as the ''employment service" or "job service" by professionals in 
the employment and training system. However, among CETA program participants an<^ 
the public generally, this office may be referred to as the "unemployment office." 
On a mail questionnaire, it is advisable to" use both alternatives. 



Example 12 



"How did. you find out about the first job you held 'after leaving the CETA 
program?" 

Telephone: 1 B Employment Service/Job Service 

Mail: 1 - State Employment Security Agency (Unemployment Office) 



ERIC 
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B. Tracking Participants 1 Post-Program Activities 

The issues of wording and format are general considerations in questionnaire 
design, but a more specific concern to prime sponsors involves the most efficient 
and effective method of tracking and recording post-program labor market and related 
activities. This section will discuss several approaches to documenting how a 

r A 

respondent's time was allocated over the post-program period. A summary of three 
techniques for tracking activities over the follow-up period is presented in Table 
3-2. 

To obtain data describing terminees 1 activities continuously for the entire 
follow-up period, it is necessary to determine the total number of weeks in which 
the individual was engaged in each of the following: 

o Working, 

o Attending School, 

o ~ Attending a Training Program, 

o Serving in the Armed Forces, and * 

3 Not working, attending schoqfVor training, or serving in the 
Armed Forces. 

These five categories do not distinguish between the weeks a participant was 

6 

unemployed or was out of the labor fferce. Generally, survey respondents have a 

difficult time distinguishing between being unemployed and being out of the labor . 

force. 'As a result, it may be more practical to simply identify the weeks the 

» 

respondent was not working . 

s The first method for tracking ppst-program activities is the fastest method; 

the interviewer asks the respondent to estimate the iivmb^r of weeks spent in each 

of the activities noted above. This approach requires ajninimal number of questions 

and is therefore probably the simplest and leats-t costly data collection method. The 

key drawback is that respondents are generally not able to give consistent estimates 

* . 8 

of the number of weeks they spent in different activities. 
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Table 3-2: Techniques for Tracking Post-Program Activities 



Technique 

1) Asking respondent to 

estimate number of weeks 
spent- in 'different activi 
over the entire follow-up 
period 



Advantages 

o Simple 

o • Requires few 

questions 
o ^ Takes the least 

amount of time 



Limitations 

o Difficult for respondents 
to estimate weeks spent 
in activities over the 
entire follow-up period 

o Respondent's estimates 
are often inaccurate 



2) Asking respondent to . 
estimate n umb er of weeks 
spent on each job held 
during the follow-up period 
and weeks not working 
between jobs 



Easier for respon- o 
dent to estimate 
weeks on ea£h ( job 
than total weeks* o 
working 

Gathers job related 
information simply 
for those respondents 
with one* or two jobs 
Leads to more accurate 
information than the 
first technique 



Very confusing for inr 
dividuals with more than 
two jobs 

Still rlHjgires respondent 
to estimate weeks on the 
job 



3) Continuous. Work History Table o Provides organiza- o Requires more time 

tional tool for and training for inter- 

sequentially tracking viewers to learn to use 
and recording all r the table 

post>program 
'activities from date 
of interview backwards 
to date of termina- 
tion* * 
o ' Relies on estimates of 
dates rather than 
number of weeks 



- . J 
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A second approach is to collect information about each individual job held in 
th^ follow-up period. Attention is focused upon the, length of employment, the details 
of each job, and time spent between all jobs* Because it asks the respoitdea£_for^ 
information job by job, this method may result in more accurate data than the first 
method. Respondents may find it easier to estimate the number of weeks on one job 
than the t0 &^ weeks on all jobs in the follow-up period. 

There are two disadvantages of this approach. First, for individuals who have 

held more than two jobs since leaving CETA, the interview may become confused as they 

attempt to sort out the number of weeks spent on each job. Second, this approach, 

K 

like the .one mentioned above, requires that respondents estimate the number of weeks 
spent working or not working which has proved difficult in the past. 

A third technique for tracking participants 1 post-program activities is to use 
a Continuous Work History liable. This table, currently being utilized in the 
Continuous Longitudinal Manpower Survey, was designed to address some of the short- 
comings of the approaches already presented -by providing an organized format for 
cataloging post-program activities* An example of this format is provided in 
Figure 3-1. 

To avoid confusion over exactly when the post-program period began (the date 
of program completion and the official date of termination may not be the same), the 
interviewer begins by asking respondents about their current activity. This is 
followed by a . brief series of questions which gathers summary information about the 
activity. If the respondent is working, attending school or a training program, the 
interviewer asks for name of the company or school, and where it is located. Those 
who are working are also asked to identify their job title. Finally, respondents , 
(including those who were not working 'or attending, a school or training program), 
are asked when they started the activity, when it ended, and what they did just 
before that particular activity, The series of questions is then repeated; this time 
directed at the activity inmediately before the one just described.. In this manner, 
the interviewer can trace all the respondent's post-program activities ' from the time 
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1980 

S M T W T r s 

APPX 

1 2 3 4 5 
b 7 8 9 10 11 12 
13 14 15 IS 17 18 19 
20 21 22 23 24 25 26 
27 28 23 30 
MAY 

1 2 3 
4 5 6 7 8 9 10 
11 12 13 14 15 16 17 
18 19 20 21 22 23 24 
25 25 27 23 29 30 31 
JUNE 

1 2 3 4 5 6 7 
8 9 10 11 12 13 14 
15 18 17 IS 19 20 21 
22 23 24 25 23 27 28 
29 30 
JULY 



3 4 5 
10 11 12 
17 18 19 
24 25 26 
31 

1 2 
7 8 9 
14 15 16 
21 22 23 
23 29 30 



1 2 

6 7 8 9 

13 14 15 16 

20 21 22 23 

27 23 23 30 
AUGUST 
3 4 5 6 
10 '11 12 13 
17 18 19 20 
24 25 23 27 
31' 

5c?TLM3ER 

1 2 3 4 5*6 

7 8 9 10 11 12 13 

-ms is-tns i9-20 

21 22 23 24 25 25 27. 
23 29 30 
OCT03I2 

12 3 4 
5 6 7 8 9 10 11 
12 13 14 15 15 17 18 
19 20 2 1 22 23 24 25 
.25 27 23 29 30 31 
NOVEMBER x 

2 3 4 5 6 7 8 

9 10 11 12 13 U 15 

16 17 18 19 20 21 22 

23 24 25 26 27 28 23 

30 1 
DECEMBER 

1 2 3 4 5 6 
7 8 9 10 11 12 13 

14 15 15 17 13 19 20 
21 22 23 24 25 26 27 

28 29 30 31 



1981 



T F S 



S M T W 

• JANUARY 

1 2 3 
4 5 6 7 8 9 10 
11 12 13 14 15 16 17 
, 18 19 20 21 22 2i 24 
25 25 27 28 23 30 31 

FE8*UARY 
1 2 3 4 5 6 7 
8 9 10 11 12 13 14 
15 16 17 is 19 20 21 

22 23 24 25 26 27 23 

MA°XH 
1 2 3 4 5 6 7 
8 9 10 11 12 13 14 
15 15 17 18 19 20 21 
22 23 24 25 26 27 22 
29 30 31 



Figure 3-1: Continuous Work History Table 



3EFOFE STARTING * (circle the date the terainee ended the CETA prograa end today's 
date on the calendar at left. Work backwards on the calendar 
covering the entire tias period as you fill out the chart below) 

SECTION II: StMyAHY post-fpocram experience 

"How t would lUfie to find out what you have been doing since ending your CETA progran, that 

is between (date ended last program) ; and now. I would like to 

start by asking you about what you are doing now, and then ask you a series of questions 
about your activities during the past few months. 



6A. "Which of the ^following activities are you doing currently?* (Head 5 Activities to 
Respondent) 



Activity Code 

1 
2 
3 

4 

5 ' 



Activities 

Working * 

Attending School 

Attending e Training Progran* 

Serving in the Arsed Forces . 
Hone of These 



; GO TO QUESTION 7 (In, Chart) 



-GO TO QUESTION U (In Chart) 



6B. 



"Before that, which of the following activities were you doing?" (Read 5 Activities 
to Respondent) 



Activity Code 



Activities 
Working 

Attending School 

Attending A Training Progran- 



:GO TO QUESTION 7 (In Chart) 



:00 TO QUESTION U (In Chart) 



4 * Serving in the Amed Forces — 

5 - None of These _ 

j _____ : — 

( Brgeyr QUESTIONS 6B THROUGH 12 , AS APPLICABLE , UNTIL THE ENTIRE POST-PROGRAM PERIOD 
IS ACCOUNTED FOR) 



Activity 
Code 


7 .'What ifS the 
Company/ 
School, 
rxaoe? 


8. Where is it 
located? 


(If working) 
9. Whet does 
this 

company 

do? 


(If Working) 
10 ."What is/ 
was your 
job title? 


LI. When 
did 
you 
begin? 


12. When 
did 
you 
stop? 
(GO TO 
Q. 6B) 


of 

Weeks 


1) 












'present 




2) 
















3) 
















4) 








* 








5) 
















6) 
















7) 


• 














8) 

















13. Are any of these jobs "CETA jobs"? . 

Ifjresi Which ones? (Circle row number) 

If : WORXIKC AT TIME ENDED PROGRAM ! GO TO SECTION IV 

NOT WORKING AT TIME ENDED PROGRAM i GO TO SECTION III 
NO JOB SINCE LEAVING CETA: GO TO SECTION VI 
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of 'the interview back to the point of departure from the CETA system. 

A key feature of the Continuous V/brk History Table is that respondents are 
asked for the dates they began and ended activities rather than the number of weeks 
they were in the activity ♦ Respondents may not always remember the exact dates, i 
but can usually recall the month, and whether or not it was the beginning or end 
of the month. By tracing, time periods of activities on the calendar attached to 
the table, the interviewer can readily identify gaps in activities or assist respon- 
dents in remembering a date. The table also collects basic information more 
activities than employment. 

The Continuous Work History Table initially requires more time and training 
for interviewers to become familiar with it, and may seem more imposing at first 
khan other approaches. However, it can provide the most organized and accurate • 
data on participant post-program activities of any of the three methods. 

The Continuous Work History Table is generally considered too complicated for 



use in a mail survey. Another version of this* approach is shown in the mail survey 
questionnaire ik Appendix D, and consists of a page with a full calendar on which 
respondents can circle the days spent in each type of activity^ Tftis calendar can 
be accompanied by a completed example which can serve as an illustration. 

With the use of the , Continuous Work History table, prime sponsors can system- 
atically trace how participants have allocated their time during the post-program 

period. It will also be "necessary to elicit t select detailed information on the 

10 * * 

nature of their activities. One approach is* to collect detailed inf ormafej&Dn^about 

V 

each employment and non- employment activity during the follow-up period. While 
this approach is the most comprehensive, it is also the most complex to administer. 

Another approach is to focus data collection strictly on participa nt^ first 
and last or current post-program labor force activities. While this appfoach pre- 
cludes the opportunity to measur^ the non-economic performance, of programs, it does 
collect data on jobs which were held at significant points during the follow-up 
period. Information on the first job held immediately after leaving the program give 
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some measure of short-term benefits of participation. The current or last job held in 

the follow-up period provides some indication of benefits gained over a somewhat 
11 • 

longer time period • 

A third approach is to focus detailed data collection efforts around i^he first' 
and current or last job, but >also include a minimal set of questions regarding any 
other employment and non-employment experiences * Although this approaches not 
simple as the one above, it provides for both a more accurate estimate of post- 
program earnings and an assessment of the non^economic aspects of program performance. 
Appendices C and. E contain follow-up questionnaires which correspond to the last two 
alternatives. 

C. Constructing the Questionnaire 

There are five broad concerns in questionnaire design w^ich will be discussed 
in this section: 



o The grouping of questions into sections; 

o The sequencing of sections; 

o * The writing of the text; 

o The coding of responses to questions; and 

o General issues in the format and layout of the questionnaire. 
1. Sequencing of Questions 

X)ncC**u£s£ions have beeg. written in a clear and concise fashion, . they must be 
arranged so that the intei^sw can proceed smoothly and the respondent can focus on 
the appropriate areas of concern or interest. The arrangement of t}>e-^estions is 
also important in.that it may help to stimulate the respondent's memory of a parti- 
cular time or event. 

A first general principle is that questions should be arranged in some logical 

* order. Usually, questionnaires are divided into sections by topic. For instance, 

<• . * S 
' the questionnaire in .Appendix C is divided into the following sections: 



s , 
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I. Respondent's View of the Program 

II. Summary. PGSt-Program Experience f 

III. For Respondents Who Were Not Working Prior to the First Job 
After CEXA 

. IV. Information on the First or Only Employer After CETA 

V. Information on Current/Last Employer if more than one Employer 
, After CETA 

VI* For Respondents Who Are Not Currently forking 

Second, a series of questions which covers^activities ..spread over a long period 
of time can be arranged in chronological or reverse-chronological order. Many people 
find it easier to remember' things in chronological order. However, when the inter- 
viewer is asking questions about activities which have occurred over a long period of 
time, it may more effective to help respondents remember the details of their current 
activity first/ and then work backwards* through the rest of the follow-up period. 
This approach was adopted in^Tfce Continuous Work History Table* ^ ^ 

Another way of organizing c^ujest'ions is sometimes referred to as "f unneling" , ^ 
Typically, this refers to arranging the questionnaire' so that more general questions 
are asked first, followed by questions of increasta^Ecif icity . By asking ques- 
tions in this order; the interviewer can graduallyH^row the respondent's focus. 

2. Sequencing of Sections 

When arranging the sections in a questionnaire, several points must be^ kept ifc 

mind. All questions of a sensitive nature, such as public assistance status, or- any 

questions that 'the respondent may find particularly difficult -'to answer, should come 

13 

later in the questionnaire. Once respondents begin answering some questions with 
minimal trouble, they, are more likely to finish all the questions. It may also be 
' helpful to ask the participants' views and opinions of the program at the beginning 
•of the questionnaire. Thus, the interviewer shows that the respondents' views are 
important and by doing so many enhance their sense of involvement. 
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A final point is that sections must be organized so *hat the interviewer is 
able to find the applicable sections quickly and without shuffling back and forth 
through the questionnaire. This is of particular concern on a mail questionnaire 
which must be organized to allow the respondents to locate the appropriate sections 
without assistance. 

3. Text of the Questionnaire 

In addition to questions and responses, a substantial amount of other text is 
necessary within a survey instrument. This text, is of two types: script, which is 
read by the interviewer to the respondent, (or by the respondent if part of a mail 
questionnaire); and instructions, which direct the interviewer (or, the mail survey 
respondent), in how to proceed through the sutvey. Generally, the script provides 
the interviewer with a standardized means for moving the interview from one set of 
questions to another. It is very important to standardize the script phrases as 
variation in individual interviewing techniques can introduce bias into the responses. 

Script phrases provide a consistent meani for interviewers to introduce them- 
selves and the purpose of the interview, and to stress how "valuable" the respondent s 
cooperation, will be. Script phrases also enable the interviewer to shift the respon- 
dent's focus from one topic area to another, as in Example 13. 

Example 13 m , 

"Now I would like to ask you a few questions about the first job you held 
after leaving the CETA program. 11 

Phases such as that in Example 14 can also create a sense of Importance for a 
set of questions. 

Example 14 \ ; i , \ — 

"This next group of questions is very important. Please think carefully 
before giving your answers." 




v ( 



In addition, script phrases can help the respondent remember what was discussed 

* 

previously, which may help to answer the questions in a particular section. Th^ 
phrase in Exaqple 15 repeats what the respondent said previously and also translates 
the length' of time into a number of weeks* 



Example 15 



"You said that between leaving the CETA program and ^you,did not 

work:. I would like to ask you a few questions about that -week 

period." . * 



The other type of text is* instructions which are necessary for several reasons. 
First, instructions should be readily available to interviewers in case they become 
-3 confused, or are confronted with a particularly unique circumstance . Second, in- 
structions provide informatipn about which sections of the questionnaire are applic- 
able to the respondent, and direct the interviewer to the next section. lAstructions 
— ^a-r-e-par-t-icularly important on a mail questionnaire as they must provide for unassisted 
completion of the survey instrument. ^ ' 

To insure that instructions can be easily distinguished from the script of a 
questionnaire, several approaches are available. Written instructions can be put in 
parentheses, written entirely in uppercase or italics, or can be underlined. Another 
technique is to use symbols such as stop signs or flags to indicate what to do next. 
If a prime sponsor has the graphics capability, questionnaire instructions can also 
be color-coded. 

-4 

' One set of instructions of special concern is the "skip pattern". Usually, not 

all questi6ns in a survey apply to all respondents. The "skip pattern" instructions 

indicate which questions are not applicable and should be "skipped". They also 

direct the interviewer to the next applicable section. Skip pattern instructions 

i 

should keep the interviewer or respondent moving ahead in the questionnaire, not 
, leafing back through partially completed pages. In addition, skip pattern instruc- 
tions are needed at the end of each section and after sorting questions such as those 
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discussed earlier in this section. Such instructions are often written as a "go to" 
statement. Arrows can also be used to provide a sense of movement or to actually 
point at the next question as in Example 16. 



"Are you currently working?" 




> r 


<4 


Yes (CONTINUE) K _ 


No — 


^{GO TO Q. 14) 




4/ 


v — ■ 







Mail questionnaires need extremely clear skip pattern instructions as the 
respondent must be able to follow them without becoming lost or 'frustrated.^ In- 
structions on a mail questionnaire may need phrases a bit more specific than 
"GO TO Q. 14", such as those in Example 17. 

Example 17 ^ - . 



PLEASE CONTINUE WITH THE ^EXT QUESTION. 
PLEASE GO TO PAGE 4, QUESTION 16. 



On a mail questionnaire* the distinctions between script, instructions, and 

skip patterns are ifot as pronounced as they are on questionnaires read by an inter- 

« 

viewer, because all are read by the respondent. However, questions and locations 

for responses must be clearly identified to minimize the chance that respondents 

« 

will miss a question* or put their answers in* the wrong place. 
4. Coding of Responses 

When designing the questionnaire, prime sponsors should keep in mind how they 
will organize tne data for processing purposes. If no computer facilities are 
available, hand tabulations can be made directly from the completed survey instru- 
merits. If the data will be entered into a computer, they must be coded. 

There are two primary methods for coding questionnaire responses. The first 
is to transfer all data on the questionnaire to coding sheets formated to resemble 
the layout of a computer "punch" card. The data are then entered into the computer 
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' ix<)tt the coding sheets.' This procedure requires that the information from the 

questionnaire b&, transferred twice, -once to. coding sheets, ^n£ then to the computer, . 
* . increasing th&p'otfehtial for misreading and miscodlpg the data* fhe alternative ' is 
^ ^ 't^Unelude the coding format. on the questionnaire itself .\ In this ca$e„ data'.can*- 
V be etttered^directiy into phe computer from' -the questionnaire, eliminating*, tfte 7!§ed * • 



for an intermediate step. 



. * * 4 
i it.- 



^^^^^ 
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17. "H6y di* you: find about thisf,. j ob? " { : 






\33' 




'...17- 








•01 « CfcTA Agency 


*• 05 


* Friends' or Relatives 


V ■, Oi » Employment 'Of fice/Job Service 




» Answered Newspaper Ad 


03 ■ Private Employment Agency 


07 


=*, Other - 






04 + Contacted Employer Directly 

i 











Example 18 is an ex^rpt from the questionnaire in Appendix C showirfg one way 
this can be accomplished. The interviewer cart enter the appropriate code number into 
the boxes at the right margin.' The coding boxes are numbered to* match the questions 
to "minimize confusion, and they are numbered ov^r the top to correspond with the 
column numbers on a standard computer card. 

. Coding techniques vary depending on the type of question. Fixed responses can 
be given numbers which can then be filled Into the appropriate coding boxes. 0pen-v 
ended questions whieh require numerical responses, such as wage rates, are easy to 
fill in^o a s£fc of coding bdxes. " However,, open-ended questions, such as "What do you 
think was the be^t thing about the CETA program? 11 , cannot have predetermined ' 
responses.. For these open-,ended questions, individual responses can be -recorded tn 
a list as the interviewing continues. Before adding a response, the interviewer 

' should check to see if it is already listed. When 'the responses on this list - ; p 
numbered, a coding appendix is created which ca t n be* used 4 to provide codes, for open-^ 

* ended questions. % 
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5. Formal and Layout of the Questionnaire 

There are a few general issues which are important in questionnaire layout. 
First, there should be ample room for writing answers to open-ended questions, or 
for fciaking notes in the margin. On a mail questionnaire, extra space can make the 
questionnaire look less complicated. Second, prime sponsors can have questionnaire 
formats reduced in size that more will fit on a page. However, reduced type can 
be mare difficult to read. It may be wise to use regular *size type on mail question- 
naires so respondents will not have any trouble reading the questions. Finally, in- 
denting questions or Sections following a sorting question may make the skip pattern 
easier to. follow, as in Example 19. 



I 



Example 19 



1. "Are you currently working?" 

V 

Yes No (GO TO QUESTION 3) 

2. "What is your job tittle? 1 ' 

3. "Are you currently attending school?" 

Yes No 



If the answer to ^Question 1 is "Yes", the interviewer proceeds to Question 2. 

*■ 

Otherwise, the interviewer proceeds to Question 3 which is indented the same as 
. • • • 

Question 1. \ ' 

IV. ORGANIZING THE FOLLOW-UP SURVEY - 

Prime sponsors have considerable flexibility in organizing a follow-up opera- 
y » * 

tion. This section discusses two key topics relating to Che follow-up operation: 
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/ ...... .. 

o The functional areas which must be addressed by a prime 
sponsor if an efficient ongoing data collection operation 
is to be established; and % 

o The selection of an organizational approach to address these 
functional areas* 

A. Managing the Follov-Up Survey 

To establish an efficient ongoing data collection operation, two key functional^ 
areas must fee addressed regardless of the organizational or operational approach 
adopted by the prime sponsor. The first includes all aspects of the data^jollection 
effort which involve accessing and using the prime sponsor management information 
system, the major source of location and contact, personal characteristic and program 
information. The second area involves all facets of coordinating and implementing 
participant follov-up surveys. 

1. Identifying and Organizing MIS Information 

The process of identifying and organizing participant MIS data has three stages. 
First; the appropriate MIS files must be identified and accessed in a timely manner 
so that surveys can proceed on a continuous basis according to the standardized 
follow-up schedule. Second, all MIS data^which may be of use in the participant 
location and contact process, such as home phone numbers and employers of job placed 
terminees, must be readily accessible to the^data collection staff. ^Third , client 
characteristic and program information needed to supplement the post-program labor 
market data captured by the participant follow-up questionnaire must be identified 
and organized. These MIS data must be organized so that they can be merged with the 
follow-up information to form one complete record which can identify participants 
with respect to their pre-program labor force history, demographic characteristics, 
programs of participation, and post-program labor market activities. This process 
of data collection and organization will depend, in part, upon the type of management 
information system available. 
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Prime sponsors that operate manual MIS records will have to retrieve partici- 
pant files by hand and arrange each participant's data set so that it can be physically 
linked with the follow-up data* This may involve the use of a separate participant in- 
formation "form to organize these MIS data elements. (A sample participant information 
form is included in Appendix F). Key data elements must be transferred from the MIS 
file to this form prior to any interview since the location and contact process hinges 
largely upon' the information contained in the MIS* In addition, it will be helpful 
for the interviewers to review the participant's character istics and pre-program and 
'program history so that questions can be asked in the most fully informed context. 

Prior to the transfer of data, instructions should be developed to indicate 
where each data element can be found in the manual MIS. This cross-reference sh9uld 
simplify and increase the accuracy of the manual MIS data transfer. In addition, a 
time schedule for the transfer must be established so that data will be available when 
follow-up interviews are attempted. If a six month follow-up period is used, the 
prime sponsor may arrange a scliedule in which the relevant MIS informati<?n is trans- 
ferred during the third manth following termination, well ahead of the survey 
schedule. 

If the prime sponsor has an aut<?mated MIS, the concerns are somewhat different - 

than those for a jnaMal system. Initially, the data collection team i&ust become 

familiar with computer printout records, lags ,in the automation schedule (time 

necessary to autpmate a complete participant file), and any data elements which are 

no/ routinely automated. It should also be determined if the procefHbill need to 

^ ^ ^^^^ 

rely upon manual back-up or supplementation, * , 

With an automated tilS, the data identification and organization task is largely 
one of computer programming* It may be convenient to identify those data elements 
to be included in the follow-up data base and write a separate file in the computer 
system. Post-program data can then be added to the file. This process must stay 



well ahead of the survey schedule* Printouts containing personal contact information 

and select participant characteristics and program dnd pre-program histories must be 

generated so that the survey schedule can proceed according to the standardized 

follow-up period* ^ 

Whether manual or % automated , MIS problems will inevitably arise with respect 

to the availability, accuracy , "and consistency of the data* Occasionally, questions 

or inconsistencies can be resolved by ^£^ag an individual case back to the MIS 

•officer, counselor or other appropriate staff person* More often, however, incon- 

sistencies or gaps in the data must serve as a basis for future improvements in 

record-keeping practices by continually underscoring the importance of an accurate 

data base and pointing out exactly where problems keep recurring* 

2. Coordinating and Implementing Participant Follow-Up Surveys 

The coordination and* implementation of participant follow-up surveys entails 

the. timely orchestration of various tasks all within the constraints of a uniform 

post-program follow-up period. Inevitably, 'the standardized follow-up period will 

vary as a result of obstacles encountered in the location' and contact process* How- 

ever, from a methodological as well as organizational perspective, it is important 

to adhere as closely as possible to the designated follow-up schedule* 
• * 
Methodologically, the follow-up time schedule is crucial since the length of 

time participants spend in the labor market affects measurements of their post-pro- 
gram income and employment experiences. Therefore, a standard follow-up schedule 
must be adhered to so that program and population subgroup comparisons can be made 
iairly, based upon equivalent post-program exposure to the labor market. Organiza- 
tionally, the uniform follow-up 'period is important so that an efficient routinized 
data collection operation can be fully established. Once serious lags devel9p in 
the schedule, a "ripple" effect tends to occur so- that each successive month it 

'becomes increasingly difficult for the prime sponsor to "catch up" and return to 

16 

the Bsroper timetable. 

V * 
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Several important tasks must be implemented within an ongoing data collection 
schedule. First, interviewer work schedules must be organized and scheduled with 
flexible hours and shifts to insure sufficient location and contact success. These 
* schedules must also .allow for variations in activity level during peak and slack 
periods, reflecting terminations which have occurred in waves or clusters rather than 
at a constant rate. 

Second, location and contact strategies must be determined and implemented ♦ 

This entails the monitoring of various contact approaches with respect to time 

i 

required, cost-ef f ectiveness , and practicality • ' This information can then be used 
t& make judgements regarding future strategies. All mailings to be sent as part of 
the location process must be scheduled and coordinated ♦ These mailings will "include 
introductory notification letters informing participants of the upcoming survey, and, 
for prime sponsors using a mail survey, the questionnaire. 

A final task associated with the operation of participant follow-up surveys is 
the implementing and supervising of all data Quality control activities* At a 
minimum, these activities should include monitoring "the accuracy, completeness, and 
internal consistency *of every questionnaire, 

A high degree of organization and attention to detail will have both immediate 
and longer-term benefits. Tn the short run, a routinized data collection operation 
.will minimize any strains on existing prime sponsor staff resourced. In the long 
run, the attention to detail will pay off in terms of a cleaner, more accurate, and 
more usable follow-up data base* 

B.» Organizing a Data Collection Operation . 

Although various "approaches can be adopted to create and organise a local 
' follow-up system, the ultimate choice depends on the availability of prime sponsor 
financial and staff resources* One major decision prime sponsors must make is 
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whether to have their data collection activities conducted in-house or through 

outside contractors, such as colleges, universities, and survey research consulting 

firms. A similar decision must be made regarding the responsibility for data 

analysis and report writing* Prime .sponsors must consider advantages and drawbacks 

of both in-house and contracted services in relation to their internal resources 

"when designing a follow-up system. 

•Conceptually, operating a follow-up system in-house increases the likelihood 
* " • 

that the operation will become a permanent organizational feature of the prime 

sponsor. The visible presence of an in-hpuse system may tend to enhance the sense 
of ownership, of and interest in the follow-up system, thus improving the chance 
that it will be relied upon for constructive inputs into the prime sponsor decision- 
making process 1 . * 

Operationally, the most significant advantage of an in-house follow-up system 

r 

is that staff will be familiar with program objectives, delivery agents, and opera- 
tional practices. This experience is helpful in the identification and resolution 
of any MIS questions, particularly those related to the nature, type, sequence, or 
duration of program activities. A level of^insight also enables the interviewer 

to assist the respondent in focusing upon the appropriate Lime period -and set of 
activities, and deci^ea^es the chance that a misunderstanding or misinterpretation of 
a question or response will occur. From an analytical perspective, an enhanced under 
standing of program objectives and operations increases the likelihood that program 
effectiveness will be accurately judged. 

A primary disadvantage of the in-house approach is that it is potentially 
disruptiv^ to existing prime sponsor operations. Not only does this activity repre- 
sent a ne^ addition to overall staff work, but it requires a level of flexibility 

^hich^y, pose an added strain on prime sponsor operatipns. Specifically, since 
mu£fr of the interviewing is done after 5:00 p.m. andi on Saturdays, staff and office 
facilities must be available at these hours. Also, the variable nature of the in- 
terview schedule over the course of a fiscal year hinders the establishment of 

^permanent. "work assignments^ 
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In contrast, an outside resource is more likely to have the flexible part- 
time staff needed to accomodate the variable schedules associated with a data 
collection operation and- have access to resources such as telephones; office space, 
and computer facilities* In addition, the overall credibility of the study is 
generally enhanced when more "ob jective" outside parties are used. Although this 
issue is most** relevant in terms of thr analysis, there is a degree^o which it 
applies to the data collection efforts as well. 

7. CONTACTING AND INTERVIEWING FORMER CETA PARTICIPANTS 

This chapter has focused upon organizational and developmental issues which 
require consideration prior 5 * to the actual collection q£ follow-up data. The 
following discussion is more operationally oriented and focuses upon the process 
of contacting and interviewing former CETA participants A Specifically, this 
section reviews techniques and strategies for eliciting accurate, useful responses 
from participants. These discussions are primarily concerned with the contacting 
and interviewing* process associated with telephone and personal Visit interviews, 
although the operation of mail surveys is also ^reviewed. N 

A. Locating'Former Participants ° x 

.Rates of response are affected by factors which characterize, individual prime 
sponsors;' such as their size and location, the accuracy and completeness of the 
local management information system's records, program and participant characteristics, 
and surrounding labor market conditions.- There are, however /Wrious search proce- . 
dures which generally apply to a wide variety of circumstances. Xhese include the 
vse of introductory notification letters, original calls and callbacks, local 
directories, direct mail methods, arid the use of outside resources such as employers,, 
relatives, and program operators. 17 
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1 # Introductory Notification Letters 

Before any follow-up interviews are attempted,, it is important to formally 
"notify respondents of the survey through an introductory letter. This letter should 
inform the Respondent of the upcoming interview, stress key parameters of the follow- 
up story, and highlight theVoluntary. and confidential nature of tte interview. In 
doing so, the letter should establish the credibility of the follow-up operation with 
'potential respondents and eiihance the likelihood that they will participate "in the 
survey. Two sample letters ate presented in Appendix *G. 

^ The mailing of introductory letters is also the first step in the participant 
location process. It provides a mechanism to both judge the accuracy of participant 



a 



(Jdresses on file and facilitate future search procedures. First, letters returned 



\ 

as undeliverable' identify the first group of - participant^ who will be difficult to 
locate. Second, when coupled with use of the U.S. Postal Service address correction, 
service, the notification process can insure that all fotwarding addresses are 
obtained for those individuals who have moved since enrollment. Subscribers to this 

4 

service receive a card from the Postal Service noting the new addVess and the date 
the addres$ change was submitted. The' subscriber is only charged for successful - 
tracking attempts. Third, business reply postcards can be included with the intro- 
ductory letter, asking terpinees who received the letter^ to fill in their current 
address and phone number and return the card. A sample business reply card- is* shown 
in Appendix G. * 
2. Original Calls and Callbacks 

Generally, the initial interview* call shou^Ee^attempted between 4:00 p.m. 

and 8:00 p.m., a time when many respondents are likely to be arriving home from work. 

For those contacted, the interview should be conducted immediately. If it is not 

J 

convenient for the respondent to complete the interview at that time, arrangements 
should be made to call again at a more convenient time. When terminees are not 
initially contacted, they should be called b*ck for a few executive evenings. If 
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contact is still not made at that point, --calls should be made at various times during 
the day. If a participant works the same hours each day, attends class, or has a 
habit oi not being at home during particular hours, callbacks which occur at "the same 
time each day will rarely lead to a contact. After five to ten days, an alternative 
contact approach should be attempted. 
3* The Use of Directories 

Perhaps the most useful alternative approach is the use of directory listings 
which' 'are available in many forms. 

o Local directory assistance usually has current telephone numbers, 
although it may take as much as a week for a changed or new. 
number to be registered with directory assistance. 

o The local telephone book may list other individuals with the same 
l^ast name. When practical, a call to the other numbers listed 
under the same last name may yield new information about a 
difficult-to-Gontact participant. 

o - There are directories which cross-reference individuals by name, 
address, and telephone number. These references can be used to 
both update or check a telephone number or locate neighbors who 
may have useful information. They'are generally published 
annually or biannually and are, available from private .firms on 
a commercial basis. , , % 

me^hc 

4, Direct Ma il Methods " 
A direct mail method can be used for any participant , f#r whpiii#fc£ie initial 
notification letter was not returned, but for whom telephone contact cannot be made 
due to unpublished, changed, or disconnected telephone numbers , In such cases, a 
second letter is sen% emphasizing that interviews are underway but that, to date, no 
accurate telephone njciber for the respondent has bteen found. This follow-up letter 
should refer ta^^hjp^'rst letter and restate the purpose of the interview and its 

confidential nature; It should include a business reply postcard requesting a 

* 

current name, address and telephone number, and a -convenient time to conduct the 
interview. The letter stfould "also provide a telephone number which can be called 
to arrange or conduct an interview^ and should note the hours when return calls can 
be made and whether collect calls will be accepted. 



Prime sponsors are advised to use all these methods. 



5. Other Information Sources 

In addition to callbacks, directories , and direct mail, several other search 
procedures are available. These fall into the following general categories: 
o Using job placement and othet post-program information, 
o Contacting friends and relatives, and 
o Using prime sponsor records other than MIS files » 

For job placed* terminees, the local MIS can be used to obtain the name, and 

telephone- number of their employers* By contacting the firm, it may be possible to 

speak with the participant to arrange an interview, or to leave a name and telephone 

number for the 'Participant to call* Even if the participant is no longer employed 
* * v ' 

' with the firm, the personnel office or another employee may have some information 
about how the participant can be contacted with a forwarding address of either a new 
residence or a new job. When appropriate and available, schools or training programs 
listed in the MIS can also be contacted either to arrange an interview or to obtain 
a current name, address and telephone number". 

If the prime .sponsor routinely records an emergency telephone number on the 
participant's MIS record, it may be possible to contact friends or relatives who can 
help in the location process. They may know a new phone number*, times the individual 
can be reached at home, or even a new name or a spouse's name in the event of a recent 
change in marital status^ 

Prime sponsor records other than the MIS are another potential source of con- 
tact information. For instance, the financial records unit may have -the latest 

addresses, especially when the j>rime sponsor -mailed a final stipend payment after the 

* 

particiant left the program. In addition, the subcontractor through which the par- 
ticipant received services or a counselor may have a current address. 

\ 
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The choice and pursuit of various location procedures must always be sensitive 
CO issues of privacy and confidentiality . For instance, if participants have given 
an emergency telephone of a friend or relative, all. other equally viable approaches 
should be tried first. While cooperation is generally not a problem,* respect for 
the privacy or the participant, friend, relative or employer is a concern which the 
prime sponsor must clearly convey to the data collection staff. * * 

B. Conducting the Follow-Up Interview 

The quality of follow-up data depends partly upon the training and experience 
of staff interviewers. Therefore, it is important for the prime sponsor fo provide 
a level of training to insure that interviewers are well versed and comfortable with 
standard interviewing techniques. As part of this training, new interviewers should 
listen to experienced interviewers conduct actual interviews, and then conduct motk 
interview's. These mock interviews should be structured to include various combina- 
tions of post-program activities, different 'levels of- respondent cooperation and 
understanding, and terminations from various programs. 

Although most of the interviewer training must take place in advance of the 
first phone call, an ongoing training' component should also be established.. This 
will^allow interviewers to share anecdotes and discuss, techniques so that all may 
benefit from increasing levels of experience. 

Some basic interview techniques and approaches which should^ be routinely in- 
cluded in interviewer training are: 

o Getting the participant to respond, 

o Using the survey instrument properly, 

o Listening to the respondent, and 

o Probing for a more accur<ate^or complete response. 
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1 4 Getting the Participant to Respond 
, Althj^gh experience indicates that most terrainees are willing to be interviewed, 
there are several instances In which they may be reluctant to participate in a survey. 
The interviewer* s response will depend upon the specific situation. First, the par- 
ticipant may have £een contacted at an inconvenient time. If so, the interviewer 
should arrange another time. to complete the interview. Second, individuals may bfe 
confused about their CETA experience, especially if they participated in short-term 
job search assistance or OJT programs which they may not have known were CETA 
affiliated. In these instances, the interviewer, using the specific programmatic 
profile on the participant's MIS records (exact program name, dates of participation, 
etc.), can attempt to jog the respondetiF 1 ^ memory by piecing together the individual's 
experience. 

Some participants may be hesitant for less specific reasons. In these cases, 
interviewers should explain three points concerning the interview. First, they should 
stress that the interviews are generally brief. If a questionnaire such as the one in 
Appendix C is used, on average, between 7 and 12 minutes is required for completion. 
Second, interviewers can note that all information is confidential and used onlyllLn 
statistical summary form, not in a form that can identify specific individuals. Third, 
interviewers can explain why follow-up studies are necessary to provide a basis for 
program improvements. » - '* 

2. U^ig the Survey Instrument Properly ' 

Generally, questions in the survey instrument should be straightforward and not 
subject to interviewer bias, or misinterpretation by the respondent. They should be 
asked exactly as worded to insure both the desired focus of the response and con- 
sistency across interviews. Efforts should be pade to avoid hinting at presumed 
responses and making offhanded comments that could influence ^respondent ' s answer. 
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In addition to using the proper wording for each question, the interviewer must 
properly use all "applicability criteria 11 . A questionnaire designed to record a full 
range of post-program experiences contains more sections and questions than may be 
applicable to any one respondent. Interviewers need to learn which sections apply to 
various situations £o insure thaf all pertinent data is captured. 
* ' 3. Listening to the Respondent 

The importance of continuously listening to the respondent cannot be over 
emphasized. Interviewers must listen to respondents even while recording a response 
because even a brief statement or comment may indicate that the participant misunder- 
stood the question or that the interviewer misunderstood the answer. Additionally, 
interviewers should not interrupt respondents except when they stray from the topic f 
of the question. An attempt ^by the interviewer to paraphrase or summarize an in- / 
complete answer can lead to inadequate leyels of detail and inaccurate data. / 

4 # Probing for a More Accurate or Complete Response / 

When the respondent's answer is not cle'ar, or indicates that the question' was" 
misunderstood, further probes are necessary to obtain accurate answers. Several 
general approaches to probing are discussed below. 

/ 

o Repeating the Question ' f 

* » » 
Oftei the respondent may not understand a question, may misin- 
terpret it, may not be able to. decide upon an answer, or may, 
w^ndrer off on an unrelated topic. In such instances, the first 
and*most effective step is to repeat the question. Many 
respondents may not have fully heard the question the first 
time' or may have missed the question's emphasis. Simply Repeat- 
ing .the question may be all that is necessary to obtain a more 
accurate response. 

e 

o Repeating the Response x 

ft may be useful f or, the interviewer to adopt the habit of 
repeating the respondent's answer word for word as it is 
recorded. This acts as a check so that the respondent^ knows 
* precisely what is being recorded, and provides an opportunity 
to validate or double-check the answer, In cases where the 
answer is clearly off track, simply repeating the answer in a 
neutral, unbiased ^manner may help the respondent sense the in- 
appropreateness of the response. A 
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Probing for Further Clarification 

• • * 

When repeating the question or the response does not seem to 
clarify the issue* further probes are required. Such probes 
should indicate that the interviewer is interested and desires 
more^ information. Far instance: 

''Could ypV tell toe more about what you are thinking?" 
"Wouid you tell me'what you have in mind?" 
"What do you mean?". 

"Why do you feel that, way?" « • 

"Which would be closer to the way you feel?" 

The interviewer knows the objective of the question, * the respon- 
dent dcfes no,t. Successful probing of the sort described here 
requires that the' interviewer 'recognize when a response does not 
meet the objective of the question. 

Pyobtng for Further Acsuracy aud Precision 

Instances may arise when _i't is apparent to the interviewer that 
a more precise response may be possible through' the use of prob- 
% ing after the initial Question and answer. The following con- 
versation is a brief example pi this approach: 

: < 
% 

Question: "What wa§ your starting hourly wage rat;e?" 

Answer: "Oh, about $4.00." 

*Probe^: "I see. Woul4'you say it was more than $4.00 or 

* less than $4.00?" 

Answer: '"Less than $4.00;" 

Probe: "Well do you think it was 'more than $3.75?" 1 

Answer: • "Oh yes, it was $3.90." 

Interviewer: (As $3.90 is being recorded) "Oh, O.K., Very Good." 
$ 

Silence or "I Don' t Know" Responses 

The intervieweOshotf^d not assume that when a question is met with 
silence or by »fche respondent saying "I don't know", that the 
respondent in fafct does not know. There are' several possibilities 

to be qonsidered ifc Such a situation. v 

. ^ » ' 

o The respondent dpes not understand the question, 

o The respondent requires/ some time to think and says* "I don't 
know" to buy time,' or\ 

o The* respondent has an answer to the question but does not' 
want to give it and wishes to evade the question. ■ ' 
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* In these cases, t% may only be necessary to repeat the question, 

. offer more time to think about it, or offer to return to the 
question later; If it becomes clear tha.t the respondent does 
not want to ansver, the interviewer must quickly determine what 
level and type of probing may be constructive without alienating 
the respondent. In such circumstances it may be appropriate to 
remind the respondent of the confidential nature of the inter- 
* view. * «• ~ 

' VI. QUALITY CONTROL AND DATA PROCESSING 

For the telephone/personal visit surveys, each interviewer should thoroughly, 

I 

review the questionnaire imnediately after the interview i$ completed. For open- 

<% 

ended questions, ^ code will have to be assigned to each response. The accuracy 

of this cpding can be* increased if it is done while the interview is still fresh 

in the interviewer's mind.- Even for fixed response questions, a review of the 
* 

responses soon after the interview makes it easier to detect any errors. Another 
advantage of. immediate checkings is that, if inconsistencies exist or questions 
arise, the respondent can be called again for clarification. \ 

' Once the questionnaire is correctly codeS, it should be .reviewed by an in- 
• dividual in a supervisory position. This second stage of checkings should focus 
' ^ on the overall COTpleteness^^^T'^internal consistency of the questionnaire. There 
<^-^;^". , are various c^S^tency checks' which can be made on a questionnaire. For instance, 
the dates began and ended jobs should all be consistent, the'number of jobs held in, 
the follow-up period should be the same as the number of jobs listed, and the start- 
' ing and final hourly wage rates on any job should bear some relationship to each 

other. It is also crucial to check and insure that ail coding boxes^re, filled in 
' and proper use made pf the "missing value" codes to distinguish a question which is 
"not* apV3-icable n from one in which the respondent "does n6t know" or "refuses to 
answer." ' * • • 
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Another issue of concern for telephone and personal visit interviews is 
verification that interviews were actually conducted. For telephone interviews 
conducted in a centralized, well supervised location, verification should not be 
a maior concern. For telephone interviews hot conducted at a centralized location 

(e.g. at the interviewer's home), and for .personal visit interviews, verification 

* * * * * * 

of a -five to ten percent sample is recommended. » 

In a mail survey, an interviewer or staff person must review and code each 
questionnaire as it .is returned. This step provides an* Opportunity to check the 
responses for completeness and consistency prior to processing the datas although, 
practically, the respondfcnp cannot be reached , to clarify any inconsistencies. 

If the prime sponsor uses computer facilities to process and statistically t 
analyze the follow-up data, more extensive edit checks should be undertaken prior 
to using the data for report writing. Additional editing and quality control proce- 
dures can be established which take^ advantage- of the automated data processing 
capability. In addition to uncovering coding, keypunch .and data entry format errors, 
this process should also provide, range checks on the responses to each question as 
well as the more sophisticated internal "consistency checks which are too time-con- m 
suming '.to perform by hand. 
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FOOTNOTES TO CHAPTER THREE 



Tho^e who' did not respond the first time were sent a mailing two weeks 
later. 

It -is unclear if this is reflective of legitimate scheduling/ conflicts 
wh^ch could not be resolved or simply a way of refusing the Interview. 

For the' purposes of comparing the survey methodologies, a rather complex 
codipg scheme was necessary, to convert mail survey results to a format 
consistent with the telephone and pergonal interview data. Some of the 
data quality problems may have resulted from this coding procedure but 
certainly not frequently enough to account for the vast disparity. 

The cost figures do not include the time of supervisory personnel who 
were managing the etftire operation. * 

If the categorized approach were used /(Option B) , it would be impossible 
to determine, for instance, how many individuals worked exactly 7 hours 
per week. ' 

To be classified as "unemployed", an" individual is not working for pay ' 
but is looking for and available for work. A person who is f, out of the 
labor force" is not working for pay and is either not Rooking for or not 
available for work. 

The current population survey (CPS) asks respondents to recallli non- 
working members of the household looked for work during the four weeks 
innediatelypreceeding the interview week. Despite this short recall 
period, a national study recently presented evidence that the data result- 
ing from the CPS are not fully able to distinguish between individuals who 
are unemployed and those who are out of the labor force/ See: Clark, Kim 
B. and Lawrence H. Summers; "Labor Market Dynamics and Unemployment: A 
Reconsideration;" Brookings Papers on Economic Activity ; 1979: 1, pp. 13- 
60. ~ ) 

An examination of the data collected as part of the Region One Follow-Up 
System shows that estimates of total weeks do not appear to be consf stent 
with dates estimated for the same time period. Checks for internal con- 
sistency indicate that the dates are, generally more accurate. 

This table is currently being utilized by the Region One Follow-Up System 
and- is included in the questionnaire in Appendix^ C. 

This would. have already been accomplished had the Continuous Work History 
Table collected detailed rather than .overview. information ont each activity 
This approach was field tested by Westat, Inc. during the development of 
' the CLMS survey instruments, and was found to be a less effective method of 
gathering data. 



-77- 



86 



11. Recent experience in Massachusetts has shown that over ninety-six percent 
of more than three thousand former CETA participants contacted and inter- 
viewed six months after semination held two or fewer jobs. Thus, collect- 
ing information on only .two post-program jbbs will^both provide a virtually 
complete picture of a participant^ labor ti^rket activity in the follow-up 
period and allow, for approximation of post-program earnings. For those * 
prime sponsors interested in this approach, it is important to clearly 
specify those data which will be collected. One possible list of useful 
job related information is presented below: 

o Job seeking methods 

o Starting hourly wage rate * * 

o Final or current hourly wage rate 

o Hours worked per week * v 

o If no longer working for employer, reason for leaving 

In addition to. obtaining detailed information about the first and current 
or last job in the folfllov-up period, it will also be useful to collect 
information about the labor market activities of participants who were 
* • either not working immediately after leaving the program or who were not 
• working at the follow-up interview. The period immediately after program 
termination is of particular importance in gauging the immediate benefits 
of CETA participation. While some period of job search is expected of 
those individuals not job placed, information on the nature and extent of 
job search activity can be quite important. 

It is reasonable to expect pa'Hicipation in employment and training 

programs to not only increase the level fcf their participants' skills, 

but also to provide them with greater familiarity with the labor market 

generally. If participants are not working and not seeking employment 

immediately after program participation, this should be a matter of some 

concern to prime sponsors, particularly if such an outcome "is repeatedly 

associated with one program activity to a greater degree than ofcher 

activities. Similarly, the employment status of participants at the time 

of the follow-up interview can also be an important indicator of the - 

extent to which particular programs -result in relatively more or less job 

search activity by those who are not employed. 

» 

For those former participants who are not employed either immediately 
after leaving the program or at the time of the follow-up interview, the 
following data elements could be included on the questionnaire: 

o Total weeks spent looking for work 

o Total weeks spent looking for work 

o Reasons for no job search 
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12. See: Gorden, jlaymond L. , Interviewing; Strategy, Techniques and Tactics ; 
The Dorsey Press* Homfewood, Illinois, 1980; pp. 361-366* 

13. Far example, questions about personal or family income, or public assistance 



status 



14. It should be noted, however, that keypunchers generally have an easier time 
working from standardized coding sheets, which may reduce the propensity 
for data errors. 

15. * This case will arise in those instances whe^fe the prime sponsor has decided 

not to automate all data elements, as is typical for the details of the 
pre-program work history. 

16. If the prime sponsor trys to return to the proper time schedule, it will 
often require a major staff^effort which may be disruptive to other areas' 
of operation. 

17. For a complete discussion of location and contact strategies see: Furlong, 
Dianne, A Guide to Locating and Contacting Respondents for Follow-Up Surveys ; 

* Massachusetts Department of Manpower Development r Policy and Evaluation 
Division, June, 1981. 

18. For a more complete discussion of interviewing techniques see: Gorden, 
Raymond L^>> op, cit . 0 



r 
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CHAPTER FOUR 
ANALYZING LOCAL FOLLOW-UP DATA 




ANALYZING LOCAL FOLLOW-UP DATA * 

V 

INTRODUCTION 

The previous three chapters of this guide have addressed several issues prime 

. sponsors* should -consider when designing and implementing local follow-up sysxems . * 

Once the systems are implemented, prime sponsors will; have to decide how their 

follow-up data will be used for assessing program performance. This chapter 

addresses this issue by reviewing a set of topics related to analyzing local 

follow-up data. The three major topics covered are the foflofci^ig: 

o, Issues related to selecting measures of program outcome; 

o Alternative types of toliow-up reports and the structure 
of the follow-Up report; and 

i 

o ^ Ways follov-up can be used- to assess the longer-term 
1 performance of local programs. 

v xi. SELECTING MEASURES OF PROGRAM OUTCOMES * 

When undertaking an analysis of local follow-up data, prime sponsors must 

* 

select the most appropriate program outcomes measures. In making this selection, 

\J * ' ■ * 

it is important for prime*sponsors to consider the merits of all potential measures , 

and their relationship to both the goals and objectives of programs and local in- 
formation needs*. Decisions among alternative outcomes measures can have a signi- 
ficant .bearing on the accuracy of follow-up findings in depicting program per- 
formance^ and their usefulness for local decision-making purposes. 

Ideally, the choice o£ an appropriate measure or set of measures of program 
outcomes should depend on the objectives of the programs being evaluated. Since 
each program is designed to achieve a specific objective, it is .Important for local 
planners and administrators to identify these objectives as well as how they can * 
best be measured. In most cases, while local programs have the broad goal of in- 
creasing former participants' employment stability a*nd earnings „ it is likely that 
they attempt to accomplish this goal in different ways. For example, a program 

o - 8 °- 
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designed to serve those characterized as having ."hard core" unemployment problems 
may have as its primary objective placement into and retention of unsubsidized 
jobs, with lesser emphasis on the t quality of that employment. In contrast, pro- 
grams geared to the problems of the working poor would be more appropriately measured' 

< 

m 

by the hourly wage rates earned by participants during the follow-up period. These 
potential differences in objectives clearly indicate the importance of clarifying 
the purpose of local programs as well as the flfantoer by which their performance can 
be most appropriately measured* ^ 

In certain instances, prime ^sponsors may also have to base their selection of 
performance measures on factors other than goals and objectives. For example, prime 
sponsors may receive reports on or evidence of certain inefficiencies in their pro- 
grams. In other instances, members of the local planning council or local officials 
may- request particular types of information regarding the post-program performance 
of local employment and training programs. In these cases, it may be desirable and 
necessary tq select a set of outcomes measures designed to address 4 the concerns 
whicn have been expressed. 

The remaining portion of this section is designed to review the strengths and 
weaknesses of seven commonly used measures of post-program performance. These out- 
comes measures are: ' . 

o Placement into Unsubsidized Employment, 

o Labor Force Status in tfhe # Follow^Up Period, • \ 

o * Job Retention, 

o Weeks of Employment, * * 

o Hourly Wage Rates, 

o Average Annual Post-Program Earned Income, and 

o Gains in Gross Earned Income. % 

A. Placement Into Unsubsidized Employment 

The first outcomes measure which may be considered by- the prime sponsor* is the 
success of local programs in placing participants into unsubsidized jobs.. This 
measure is determined by the following calculation: 
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Percent of •Terminees Placed » Number of Job Placements x 100 
^ in Unsubsidized Jobs Total Number of Terminees 



The major strength of this outcomes measure is that it is consistent with a 
primary objective of the CETA Title II delivery System.- Additionally, as one of 
the Department of Labor *s performance indicators, the .information shsuld be readily 
available from the prime sponsor MIS and can be used to 'compare current with pre- 
vious levels of performance as well v as with those of other programs. 

There are, however .several shortcomings associated* with this outcomes 

measure. First, it captures information only immediately following program termina- 

• ✓ . ■ , 

tion. As such, it does not provide a basis for judging the longer-term benefits of 

2 - 
program participation. Secondly, success in generating a certain level of unsub- 
sidized job placements does not address differences in the types and quality of 
employment generated for program terminees. 

The third shortcoming is that overreliance ofrthis performance indicator may. 
discourage the provision of services to those most in need. 'Since it will be easy 
\o achieve high levels of placemertts by serving those with comparatively limited 
barriers to employment, prime) sponsors may become inclined to "cream 11 from the 
eligible pool of -applicant's and provide 'only a minimal level of services to the 
more disadvantaged segments of the local population. 

* ♦ * 

B. Labor Force Status in the Follow-Up Period . 

A second measure of post-program performance is the labor force status of par- 
ticipants at a given point in the follow-up period. Categories' which have tradi- 
tionally been used to tneasure this type of program outcome are: 

o Employed (working for pay), ^ 

o Unemployed (Not working for pay, but looking for and avail- 
able for work), and m 

o Out of the Labor Force (Not working for pay an£ either not 
looking for or not available for work) . 



The terminees 1 labor force st^tus^ at a particular point in time can be 
measured at any juncture during the follow-up period, e.g. at three, six or nine 
months. To assess former participants 1 labor force status at the time of follow- 
up contact, the appropriate measurements are: / 



Number Employed at FollovUp. Interview A 
Total Number Contacted for Interview 

Number Unemployed at Follow-Up Interview 
Total Number Contacted for Interview 

Number Out of Labor Force at Follow-Up Interview 
Total Number Contacted for Interview 

TOTAL 



/ X 100 - 
X 100 - 
X 100 - 



Percent 



Percent 



Percent 



100 percent of Population 
Contacted ' for Interview . 



Measures of post-program labor force status provide information on the success 
achieved by terminees in securing employment. In addition, this measure also pr^-r 
vides an added time dimension and a detailed perspective on those who are not working 
at a particular point in time. 

There are, however, several drawbacks associated with the use of this outcomes 
measure. First, such measures of post-program labor force status do not consider „ 
the type and nature of work performed. Inter-program comparisons of effectiveness 
based upon this outcomes measure could have limited usefulness due to differences 
in wages and skills of the jobs held. Secondly, labor force status measures reflect* * 
bnly one point in time and do not. fully represent total post-CETA employment 
experiences of terminees. 



C. Job Retention 

Another outcomes measure is job retention. Thi's measure can be defined in two 
waysr 1) as the number of weeks that program terminees remain employed on the job 
into which they were placed upon termination; and, 2) as the percentage of those 
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job placed who are still employed on the same job at the time of the follow-up, in- 
terview. This latter definition is calculated as follows: ^ 



Total Job Placed and Still Employed on 

Same Job at the Time of Contact 

Total Number of Jab Placed 



Percent of 
X 100 =» Job Placed 

Still Employed 



In contrast to the point-in-time measures discussed above, measures of job 
retention provide a more continuous longer-term perspective of program performance. 
f As such, this measure i^. quite useful for gaining insights into post-program job 
stability. 

Despite apparent merits of job retention, it does possess several limitations. 

First, the exclusive focus on job placements does not provide any insight into the 

• v 

employability of terminees who were not placed in a job or did not find a job on J 
their own upon termination. Secondly, an examination of employment patterns 
associated with only the "placement " job provides an incomplete picture of post- 
program labor market experiences when terminees have left their^ placement jobs at 
some time during the follow-up period. For example, an individual may ha^i** 1 ^ 
experienced a low level of job retention because of the opportunity to secure a 
higher paying job. The third limitation is that, lifce measures of job placement 
and labor force status, post-program job retention does not take into account thfe 
types and nature of employment opportunities available to former participants. 



D. Weeks of Employment 

A fourth outcomes measure that can be considered by prime sponsors* is the 
total number of weeks of employment during the post-program follow-up period. This 
measure can be expressed as the total number of weeks worked on all jobs held during 
the follow-up period (Equation A below), or as the percentage of the total follow-up 
period which was spent Employed (Equation B below). 



H 

Z WKS 
j-1 



i 



Where: WKS . ■ Number of weeks worked on job j 

N ■ Number of jobs held in the follow- 
up period 



MC 
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' b. Total weeks of - employment X 100 31 Percent of Total 

Number of weeks in the follow-up period Follow-Up Period Employed 

v * 

TJie strengths of this outcomes^measure are considerable, particularly in chat 
enhanced employment stability is a primary objective of many CETA programs. Unlike 
the job retention indicator, total weeks of employment measures performance over 
the entire post-program follow-up period and applies to all CETA participants , regard- 
less of their termination status ♦ 

Major drawbacks to weeks of post-program employment are that: 1) since it is 
a gross measure of program outcome, it does not measure the independent contribution 
of the program to weeks of employment; and, 2) it does not account for the 'quality 
of employment obtained by former participants^ Thus, instances of steady employment 
accompanied by relatively low wages and poor working conditions would not be captb^ed 
by this measure. 

V 

E. Hourly Wage Rates 

An additional outcomes measure which may.be used ik the average hourly wage 
earned by former participants. This* measure can capture the average hourly wage over 
the entire follow-up period or the actual hourly wage rate at a particular point in 
time. To calculate the average hourly wage rate over the entire follow-up period, 
pr*q>e sponsors must identify the wage earned by participants on each post-program j6b 

and weigh' it against the -length of time spent employed on each of the^e jobs. The 

1 

formula that would be used is: * 



^ N W ) * WKS 



j-1 WKS i 

Where: ■ Wj -Jlourly wage earned on the«jth job 

♦ WKS.I -Number of weeks worked on job j 

J 

N -Numbefc of jobs held .in the follow-up period 
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The strength of this measure is its ability to reflect the degree to which 
participation in a program has enhanced the productivity of participants ♦ To the 
extent that hourly wage rates reflect productivity, those who realized significant 
benefits . from the program would, in a relative sense, earn higher wage rates. 
This indicator is also a partial reflection of the value that the labor market 

V .6* 

places on the skills which have been acquired through participation. 

However, when prime sponsors choose to measure hourly wages at a particular 

point during the follow-up period, judgements of program effectiveness will also be 

limited since they are not based upon any knowledge about the employment stability 

of participants. While a participant may have earned a high hourly wage rate, 'it 

may only have been earned during a brief portion of the post-program^ period. A 

* 

final limitation is Chat, like total weeks worked,' measures of hourly wages reflect 
a gross program output. As such, issues arise related to the targeting of services 
and the accurate assessing -of the -independent effects of a program. 

F. Average Annual Post-Program Earned Income 

The most comprehensive measureof post-program performance available for use 
by prime sponsors is the average annual post-program earnings of former CETA partic 
ipants. Such a measure could be calcylated as iollows : m 

Y - W * H k WKS * '52/TW ' " \ 

W^efce:, 

a [ 

ilized earnings \f the participant based on income earned over 
:ollow-up period 

jrly wage rate^^rned by the* participant over the f ollgw-up 

_ ^riod, 

H - ^v^d^ehou^w/rked per wee£\by the participant over the follow-up 
period *w X t J * 

WKS ■ Total number ot\hfcks each participant is employed over the follow-up 
period 

1 TW » Total number of weeks between the date of termination to the date of 
4 interview • % 
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This gross outcomes measure provides a comprehensive measure of post-program 
performance, containing both an employment component (hours and weeks worked), and 
an earnings component (average hourly wage)* The overall status of terainees' 
earnings and employment can thus be judged using this summary ^outcomes measure. 
However, because annual earnings is a summary 1 measure , it is not possible to 
distinguish* how* different programs generated a given level of earnings. Since 
earnings are fhe product of wage rates and employment, any level pf earnings reflect 
both job quality and employment stability. 

* 

Gr Gains in Gross Earned Income 
Jf 

As a gross measure of post-program , output , average annual ^earnings does not 
identify that portion of post-program" income which would have been earned in the 
absence of participation in the program. To the extent this is unknown, ^here may 
be an incentive for prime spgnsors to serve the less disadvantaged, 'that is, in- 
dividuals more likely to earn comparatively higher infeomes regardless of their 
training. « 

One way to avoid incentives to serve the less disadvantaged Is to judge.post- 
program performance on the basis of any gains in former participants' earned in- 
come. This outcomes measure is quite desirable for local follow-up purposes since 
it explicitly reflects the basic intent of CETA and will allow prime sponsors to - 
identify the independent contribution of program participation to earnings during 
the follow-up period. Gains in earnings also takes into account the stability an^i 
continuity of post-program employment as well as changes* in participants' hourly 
wages. As such, the use of this measure should create an incentive for the prime 

sponsor to serve the more disadvantaged. 

\ ' * ' , 

From a practical perspective, .prime sponsors may encounter certain technical 

• / 

difficulties when attempting ; to measure 'gains in earned income. First, to the 
extent that pre- to post-program comparisons are used to calculate income gains, 
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prime sponsors -will have to carefully select a pre-program period sufficient in v 

length* to accurately portray the prior employment and earnings status. of former 

participants., Even resolution of this issue, however, will not necessarily 

* *• 

» • •» . , 

result in accurate estimates of income gains due to the potential influence of 

v « * 3 
'maturation 1 and changes ip*the condition, of the local economy , ^ 

Secondly, while the vise of comparison or control groups may w,ell result in 
.more accurate estinjates, such an approach will likely not be pragmatic for broad 
scale^^eal followrup efforts. Therefore, prim& sponsors will have t£ give 
serious consideration to bot^h the use of this outcomes measure as well as the 
manner by which it will be estimated. * , 

The above discussion has focused on the major strengths and weaknesses of 
alternative program outcomes measures. Table 4-1 presents, in summary form* the 
relative merits of each of the seven measures discussed above. 

III* SHAPING THE LOCAL FOLLOW-UP REPORT # . f 

There are two key issues prime sponsors will have to consider once outcomes 
measures have been selected. The first issue involves selecting the type of 
,fbllow-up reports to be prepared and used for planning purposes.^ Pripe sponsors 
can choose between descriptive and analytical reports. Descriptive -reports are 
designed to provide ' an overview of program performance. Such reports rely on any 
number of program outcomes measures and seek* to determine the uniformity of per~ 
formance across both programs and selected population subgroups. A major feature # 
9f this type^f report is that it identifies how well local programs perform 
rather than why they perform as observed. 

• Since descriptive reports are not inferential in nature, they tend^ to rely 
bn Basic statistical techniques, such as tests 'between means and contingency table 
analysis.. As'a result, these types of reports can only provide limited direction 
for determining how local performance can be improved. Descriptive reports are, 
however, easy to prepare and cdn identify those areas of the local delivery t system 

that may require improvements. 
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Table 4-1: Measures of 'Program Outcome 



Definition 

A. Placements Intq»Unsub- 
sldi2ed Employment 

o Participant placed in 
-unsubsidized employment 

B. Labkr Force Status in the 
Follow-Up Period 

o Participant is either* 

employed, unemployed or 
> • out of the labor force 



Advantages 



o Primary objective 
of program . 



Employment is a major 
focus of the program * 
Provides broader view 
of participant's status * 
than job placement; 
includes those not working 



Disadvantages 

o Not s a good measure 
of long-term program 
benefits 

o Does not address 
issue of type and 
quality of -employment 

o Point-in-^ime measure 
provides <only limited 
view of activity over 
time 

o Does not address 
issue of type or • 
quality of work 

o Does not provide 
indications as to 
why participant is 
unemployed or out 
of the labor force 



Job Retention 



o Number of weeks program , 
terminees remain employed 
on the -placement job 



D. Weeks of Employment 

o Number of weeks .worked 
in. follow-up period' 



o Employment is an important 
long-run program objective, 

o Measure of stability of 
employment* 



o Employment is important 

long run program objective 
o Measures job stability 
o Looks at experience' of 
entire follow-up period, 
including job changes 



0 Only measures work 

1 on placement job 
although switch to 
another job could „ 

r epr e s ent imp r ov emen t 
o Provides no consider- 
ation of compensation 
' per week of work 
o Provides no indication 
of employability of 
those tenhinees not 
job placed 

o Provides no consider- 
ation Qf compensation 
per week of work or 
employment * 

o Cannot reveal indepen-/ 
dent contribution 
of program to partici- 
pants' employability 



ERIC 
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Table* 4-1: Measures of Program Outcome (Cont.) 



Definition ' ' 

E. Hourly Wage Rates 

o Hourly wage rate at* start 
of ,f irsD job, average 
over follow-up period or 
at 'time of interview . 



Advantages 

o Measures the ability -of the 
participant to generate, 
aarnings for each hour of 
work 

»o Reflects value that labor 
market places on skills 
which have been acquired 



F* Average Annukl Post-Program o Measures ultimata long-term 

program goal 

o Annualized income earned . o Provides summary measure* of 
over the follow-up program, benefits 

period * 



% D is' advant ag es 

o Does not distinguish 
between possible other 
factors which may 

% affect post-program 
wages (e.g. previous , 
work experience) 

o Does not measure employ- 
ment stability 

o Gross earned income is 
a summary measure and 
does not provide an 
ability to understand v 
the source of variation; 
wages or weeks of 
employment , «/ 

o Does« not provide insight 
into which 'portion of 
the post-program income 

. would have beer\ earned 
without program particip 
t Ion 



Gains in Average Annual 
Earned Income 

o Difference between 
earnings in .the year 
prior t6 the program 
and the annualized 
earnings over the 
follow-up period 



Provides summary measure 
of; program benefits 
The change in earnings 
measures the impact of 
the program and removes 
the effects of individual 
socio-demographic charac-^ 



teristics 



o Earnings in the year 
previous to the 
program may npt be 
representative of 
long-term earnings 
and may not fully 
control for individual 
characteristics 

O'The implementation and 
use of either a control 
or comparison group, 
while theoretically 
feasible, will likely • 
not' be practical for 
broad scale local 
implementation 



• / 
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Wl)en pifime sponsors wish to gain more insight* into the performance of their, 
delivery systems, analytical, reports can be quite useful* These reports are more 
inferential in nature and rigorous in design than descriptive reports* Analytical 
reports, pend to fpcus on a detailed examination of a narrow range^of performance 
measures to determine both the" independent effects of program' participation* and 
how che performance of certain programs can be' sustained^and that o£ dfch'ers improved* 

Analytical reports may be more desirable due to their ability to provide 

specific direction for improving local programs. In 'making a selection, however, 

prime sponsors sho'uld carefully consider their specific informatiorj needs and their 

ability to use the findings from analytical reports. In cases where*prime sponsors 

have had limited experience with analyzing and using f ollow-yp data, descriptive ^ 

reports may* be desirable* Since descriptive reports are not complicated, they qan 

serve to both introduce the'sponsor^Co post^program performance measures and aid 

in a determination of the most useful types of analytical reports to produce in the 

future. Table 'A- 2 lists the major strengths* and weaknesses of these two types of 

I ' \ 

reports. 

The second issi>e to be considered is structuring the local follow-up report* 

oil s 

Follow-up reports must be prepared to show/a complete and consistent picture of the 
post-program experiences of former participants with respect to a particular out- 
comes measure. Follow-up. Reports which inclttdf a number of outcomes Measures not 
related to one another,' or which; by design, exclude certain^ypes of individuals, 
may be of limited value since °they'may ndt be capable of fully, assessing program 

performance. Prime sppnsors should shape their 'follow-up reports in a way that ^ 

» * 

'most fully describees how former participants have fared in the labor market. 

*■ 

This can be accomplished by' both tracing a particular aspect of former par- 
ticiparits 1 post-program employment and earnings experience, and conduct ing t comple- . 
mentary 'analysis of v f orihq^parti£ipants who may be excluded from an outcomes peasure, 
For example, if postjpprogrdto' hourly wage rates are selected for analysis, prime 
sponsors should examine this outcomes measure at several points ^n time to provide 

K 

• * *> 
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Table 4-2: Major Strengths and Weaknesses of- 

Alternative Types of Follow-up Reports. 

\ 



Advantages ' ~ 

4 . i 
Descriptive Reports , 

o Can provide ^ comprehensive Jovep/iew 
3 of several 'aspectsyof performance 
o Relatively easy to prepare and not , 

very time-cpnsuming ; 
o Can provide general direction for areas 

requiring improvement or further 

analysis * /* * 

o Relatively easy to reiti and a 

useful starting point for v prime 

sponsors • ' 

Analytical Reports , « 



Disadvantages * 



( 

Well fc^cusjed on a particular issue 
Capable of and designed to isolate 
program effects 

Can provide clear direction on how* 
to improve program performance 



Limited in their ability 
to provide specific and 
detailed direction fior 
program improvements 



o 
o 



Time-consuming to prepare 
Difficult to present in a 
nontechnical manner 
Require^the use of technical 
rigor 

Requires availability of 
technical expertise 

v V 
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.ute post-program 



for an assessment of both absolute po'st-program earnings and changes which may have 
occurred during the follow-up period. Also^ certain outcomes measures, such as job 
retention, may exclude all participants who were not job placed as well as the con- 
sequences of leaving a placement job for those. who were, .To address this,^ it will 
be necessary to conduct complementary analysis of the post-program employment 
experiences of both non-placements and those who left their placement jqbs prior to 
the follow-up interview. Suc# an approach would result in follow-up findings that 
could be ^linked together and, generate a more complete picture of several aspects of 
post^program employment. 

\ 1 > . ' ' ' - 

IV. AN ILLUSTRATION OF HOW TO USE AND ANALYZE LOCAL FOLLOW-UP DATA 

tfhis section of the chapter describes how f9llow-up data can be analyzed and 
used to assess longer-term performanpe of local programs. The data used for this 
illustration include over 3,600 program participants who terminated from occupational 
training and job search assistance programs funded under Title IIB of CETA and 
operated by ten Massachusetts prime sponsors. These individuals terminated ?Jfom 
programs during the March 1978 to M^pch 1980 time period', and were contacted and in- 
terviewed over the telephone on their six month post-program anniversary date. 

The major focu3 of this illustration' is on comparisons of participants 1 start- 
ing and final p.ost-prog'ram hourly wages as wel^.as their total number of weeks 
worked. For each outcomes measure, increasingly detailed and sophisticated analyses 
are conducted to provide prime sponsors with examples that can be usecf to guide the 
preparation of local follow-up reports. Additionally^since prime sponsors possess > 

a fair degree of control over the type and nature of servicesXtd be offered, the 

** / 
illustrative cqpiparisons should be useful in depicting how follow-up data can be of 

assistance in making program mix decisions. 
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A. The Programs to be Analyzed 

<• » 

The former participants who are included in^this analysis terminated from two 
types of programs* typically offered on the local level, job search assistance (JSA) 
and occupational training (OT) . " Since many prime sponsors offer these as program 
alternatives, information on their relative effectiveness can be used in local ser- • 
vice mix decisions. Thi^ section of the chapter illustrates the first step in 

* 

analyzing follow-up data and includes a description of the nature and purpose of » , 

the two programs under study. , ' ( ¥ 

1. Job Search Assistance m ~ 

Job search assistance programs are founded on the premise that the major \ 
barrier potential participants £ace in the local labor market is insufficient know- 
ledge of: job search techniques. Job^ search assistance programs generally range from 
one to eight weeks in duration and provide participants with" the support they need 
to obtain empfoyment more quickly than* t^iey could otherwise. 

The terminees included in this analysis ^received two major types of job search 
assistance sei^vices. First, approximately 85 percent received only direct placement 
types of assistance. Tt\ese participants.* were identified as the most ,f job ready ff in 
that they possessed marketable skills but were experiencing some difficuity^n secur- 
ing employment. Foi/ some, attempts were made to develop jobs and place them directly 
into unsub^idized* employment. For others who had marketable skills, but who could 

M - « 

benefit from some instruction in search skills such as interviewing techniques and' 
how to present oneself to a prospective employer, limited services were available. 
These services consisted of a very brief seminar jon where and how 1 -to find a job, 
resume writing, interviewing techniques and other related topics. ' 

^Th£ remaining 15 percent of the participants received mote intensive job search 
assistance for about four to six weeks. These individual^, while also possessing i 



marketable skills, were identified as relatively deficient in most aspects of the 
job search process • They received very detailed instruction in job search skills, 
interviewing techniques, establishing employment contacts, enumerating personal 
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strengths and* weaknesses, and identifying employment goals. • 
2. Occupational Training r * 

In contrast to JSA, occupational training programs ate longer in duration and 
* designed to provide individuals with a specific occupational skill. These programs 

• reflect the premise that participants have limited, if an^, vocational skills. 

Participants in OT progtams were offered an array of different types [of trainings 

As shown in Table 4-3, the largest proportion received training in clerical skills 

(29.6 percent). Machine arid electronics training also accounted for large shares 

of participants, enrolling 10.1 percent and 9.2 percent of *all individuals, respec- 

* 

tively, while training in areas such as computer, health, machine and transportation 

6 

related occupations was offered less frequently. *> 
In addition, to identifying the major types of services received by participants, 
it is important to examine «the nature of program participation since it more 
accurately reflects the comprehensive service strategy pf the local delivery system. 
Information on the duration- of participation^ as well as the combinations o*f programs ^ 
offered po participants can be useful for determining how exisj^ng strategies can be 
improved in the future. Table 4-4 shows 'the distribution of OT participants who 
participated in two or more activities by the activity in which they participated. 
As can be seen, the vast majority of participants received only pne service, while 
22 percent participated in at least one activity prior to enrolling in an OT program 

* (column one, 576/2594) and 7 percent in at least two activities (column two 196/ • 

y 

2594). In addition to participating in more than one OT skill area, individuals 
also participated in such pre-vocational activities as adult work experience, adult 
basic education and english as a second language. This information reveals the 
limited nature of program sequencing and may serve as the basis for identifying* 
major strengths and weaknesses .of local programming efforts. , 

The average duration of participation in. an OT program was 18 weeks. This con- 
trasts with the 6 week average length of participation* in t)\e JSA programs. Addi- 
tionally, JSA participants did not always receive full-time services. In the OT 



Table 4-3: Occupational Training Program Participants 
; By Skill Training Area 



Skill Training Area 
Clerical Related 
Electronics delated 
Health Related ' 
Metal Related 



Auto 



and Englne^EeJLated 



Food^PreparationMtelated 
Machine Related 
Computer Related 
Printing Related m ^ 
Maintenance Related 
Carpentry Related 
Transportation Related 
Cosmetic Related 
Accounting/Bookkeeping 
Other 
Total 



* Number 
769 
240 ? 
144 

77 

93 

82 
261 

66 

* 38 
54 

* ; 3 

91 
2594 



Percent 
29.6 
9.2 
5.5 
*.0 

10.1 
2.5 
2.5 
1.5 
2.1 
0.1 
0.1 
3.5 , 
23 .*4 
100.0 



J. 



\ 




Table 4-4: Second and Third Most Recent Programs of 
' Occupational Training Program Termnees 



Second and Third Most 
Recent Activities 


Second Most Recent 
Activity for Participants 
in at Least Two Programs 


Third Most Recent 
'Activity for Participants 
in at Least Three Programs 




Number 


^Percent 


Number 


Percent 


Occupational Training 


104 


l q r\ 


1 7 


8.7 


' On- the- J ob-Training 


14 


2.4 


4 


Z . U 


Adul^t Work Experience 


99 


17.2 


46 


23.5 


Adult Basic Education" 


109 


18.9 


22 


11.2 


GEO Preparation 


47 


8.1 


J 13 


6.6 * 


Job Search Assistance 'vn 


17 


* 2.9 


2 


1.0 


-i 

Assessment and Orientation 


101 


7.5 


75 


38.6 


English As A Second Language 

V 


54 


9.4 
1.0 

> 




4.1 


Special Programs * ** 


6 


5 


2.5 


Public Servide Employment 
* 


21 


3.6 


3 


1.5 


Youth Programs 


4 , 


'0.7 


1 


0.5 


JTotal 


576 ' 


- 100 


196 . 


.10(5 

•4 
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programs, however, participants attended on at least a half-time and, in some cases, 
a Sull-time basis. Thus, the 18 week average length of program attendance means 
that OT program participants receive substantially more than three times the amount 
of training services received by JSA participants. 

4 

B. Data Collection and Participant Characteristics 

The data used in the analysis that follows v have been assembled by the 
>Jassachusetts Department of >l4npower Development (DMD) as part of a state-wide effort 
to enhance the follow-up capabilities of prime sponsors. Each prime sponsor under- 
takes a six-month follow-up survey of j^cticipants who terminate from its Title IIS 
programs. Data collection procedure^, used are generally consistent across all 

sponsors and rely upon the telephone to contact terminees six months after ^ermina- 

' J 

tion from a program. Once follow-up data have been collected, each prime sponsor 
produces an analysis of post-program performance and, on a periodic basis, transmits 
local data to the state for the purpose of establishing a state-wide data base. This 

\ 

data base is used to conduct state-wide analyses of ^ the effectiveness of the CETA 
program. 

This section provides an overview of data colledfion procedures and the 
characteristics of those Included in the study population. The usefulness of the 
analysis is that it provides a basis for gaining insights into the nature of those 
participants to be studied and judging the quality and Representativeness of the 
follow- up *data. * I 

There were 6,869 individuals who participated in and \ terminated from JSA and 
OT programs. Of these individuals^, 25.8 percent participated fn JSA programs and 
74,2 percent* in OT prpgrams. . As shown in Table 4-5, there are large differences 
between the characteristics of participants in both programs*, JSA participants were 
older and presumably more experienced than the participants iii OT,' Also, JSA par- 
ticipants were more educated, as 41 percent completed 13 or more years of education 

\ 

compared to 10 percent of the OT participants. *A significantly larger proportion of 



* 

Table 4-5: Distribution of Participants by Program 
^nd Demographic Characteristics 



si' 



^ * Job Search Assistance * Occupational Training • Total 



Number Percent * Number Percent Number Percent 



AXX* PARTICIPANTS 
AGE* * 




12 

13 and over 
SEX a 



1770 




25.8 


5099 


74.2 


6869 


• 100.0 














r ( 
















. 187 




10.9 


771 


16.4 


958 


, 14.9 


% <~ 524 




* 30.5 


" 1555 


33.0 


2079 


32.4 


789 




45.9 


1998 


42.5 


2787 


43.4 


218 




12.7 


382 


8.1 ' 


600 


9.3 



8 and below 142 ' '8.0 505 9.9 647 9.4 

9- if 



280 - , 15.8 1697 *3.3 19/7 28.8 

613 34.6 2377 4jP.6 2^90 43.5 

.735- # 41.5 520 . -10.2 >i255' 18.3 



Male 980 
Female 787 



RACE* 



White 1321 

Black ' 291 

O-fcher 155 

AFDC a ^ f 

Yes t 7^ 

No ' 1697 



55.5 


* 2354 


46.3 


3334 


48.7 


44.5 

i 


2727 


53.7 


3514 


51.3 


74^8 


3106 


61.2 


4427 


64.7 


JL6.5 


1151 


22.7 


1442 


21.1 


•8.8 


821 


16.2 


976 


14.3 


'4.1 


735 


14.4 


808 . 


11.8 


95.9 


' 4364 


85.6 


6061 


88.2 



Significant at the 1% level 

/ 
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males' and whites participated ip. JSA relative to OT, and a larger proportion of 
APDC recipients was selected for 0T» icross both programs, participants tended to- 
be 25 to 44 years of age, female, white, not receiving AFDC, and classified as 
having completed at least twelve years of education* 

This pattern of program assignment should* not be surprising. The JSA activity 
is designed primarily for*those who are identified as "job ready", while the OT 
'activity^ designed for individuals who possess significantly greater ( barriers to 
employment. Since OT participants appear to be more disadvantaged, the observed 
distribution of participant characteristics across the programs would seem reason- 

y 

able. Whether, and to what extent, these differences influence the outcomes of both 
programs is an important issue and will be discussed later in the chapter. 

The ability to contact and interview these terminees varied by both their, 
characteristics and program of termination. Of the 6,869 participants, 57 percent 
were contacted and 56 percent were successfully interviewed. For JSA. participants, 
65 percent were contacted and 58 percent interviewed. The results for OT partici- 
pants s^wed a'corjCact rate of 54 percent and a completed interview rate of 51 per- ' 
cent. These differences reveal the uneven representation of JSA and OT participants 
in the 'follow-up population and may produce some £ias in the follow-up data. 

Table 4-6 shows major reasons why former participants were either not contacted 
or not successfully Interviewed upon contact. % Problems with telephone numbers - 
acquired from the local MIS accounted for ?6.9 percent of all reasons for the absence 
of fallow-up data (rows c^ie arid two in the ta^le) . In contrast, while information 
was available for 38.6 percent of the non-respondents, (top five rows in the table), 
changes in their phone numbers. and addresses prohibited a follow-up interview. That 
participants typically will respond to an interview once contacted is shown* by the 
fact that only 9.3 percet^ of these non^respondents refused to participate in the 
follow-up survey. The group of non-respondents who refused to answer constitutes 
only 3 percent of the total population of terminees. 
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Table 4-6: Reason For Not completing Interview 

(N=2403) 



Reason Not Completing Interview 
No Phone Number at Intake 
IncorrectjPhone Number at Intake 
Disconnected Phone 
^Unpublished' Phone Number 
Moved: Address Unknown 

s 

Participant ^Refused to Answer 
Language Problem 
Could No't Contact 
Total 



Number 
434 ** 
, 381 
325 
169 
435 
223 
61 
545 ^ 
2403 



Percent 

11.0 
' 15.9 
13.5 
7.0 
18.1 
9.3 
2.5 
22.7 
100.0 



The category "could not contact" is a residual classification that includes 

( 

instances which arose during the follov-Op survey but were tf5b small in number to 
warrant a separate listing* This category represented 22.7 percent -of all reasons 
for non-response and included such things as .incarceration, hospitalization, en- 
listment fS the military and^ continual absence from the home when contact -was 
attempted. Additionally; if participants c5uld not be contacted 1 or interviewed for 
more than one reason, they were also included in this category* 

.Table 4-7 shows program and -demographic characteristics of participants who 

responded to the follow-up interview . The characteristics of the participants who 

. 4 » ~ 

responded tend tb be similar ^o those of the entire population, although there were 

some differences*. For the sample of respondents as a whole, as well as for respon- 

dents 1 from JSA and OT programs separately, respondents tended to be oilier, better 

* • * I 

educated, female, white, and recipients of AFDC. These* differences may produce 

some bias in the follow-up clata, ,and should be kept in mind when judging the effec- 
tiveness of local programs. 



C. An Analysis of Average Measures of Program Outcomes. 

This section provides* illustrations of how the use of simple statistical 
techniques can provide important insights into the level And distribution of post- 
program performance-. Table 4-8 shows 0 that the hourly wage earned by terminees on 
their first and last or. most current ppst-program job was essentially equal for the, 
JSA and OT programs. The data also show that'hourly wages, are highly related to 
the age, education, sex, race, and AFDC status of^-former participants. The specific 
findings can be summarized as follows: 

o Hourly wage rates tend to increase through the middle afees 
(up' to 44-years of'age), and then decline; 

o Wage rates tend 4&> increase "as the level of education in- 
creases; and 4 • , , 

o Hourly wage rates are higher for men than for women, as 

they are 'for whites and non-recipients of * AFDC. . . ' 
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v Table 4-7: Distribution' of Respondents by Program 
and Demographic 'characteristics 



Job Search Assistance 
Number Percent 



ALL RESPOND5NTS 
AGE a 

16-19 
" 20-24 
' 25-44' . * 

45 and Ove^r 

EDUCATION * 

8 and Below v 
9-11 ' . 

12 

13 and above 

Sfo a " ' 

Male 
Female 

RACE a 

White 
Black 
Other* 



AFDC FEMALE HEAD OF 
HOUSEHOLD* 



1023 



120 
298 
448 
131 



V 145 

345 
' 469 



Yes 
No 



539 
484 



813 
145 
63 



42 
981 



28.1 

J 

12*0 
29,9 
44.9 
13. li 



6.3 
14.2 
33.7 
.45.8 



52.7 
47:3 



79.6 
14.2 
6.2 



4.1 
95.9 



Occupational Training 
Number Percent 



2616 



354 
869 
1121 
244 



216 
805 
1292 
303 



Total 
Number Percent 



lbsfe 

1554 



1632 
608 
365 



" 421 
2195 



71.9 


3639 


100.0 


< 






13.7 


474 


13.2 


33 * o 


lib / 


J £. .b 


43.3 


1569 


43.8 


9.4 


375 


10.5 




» 0 




• 




"7 "7 

7 . / 


8.3 


280 


30.8 


95 0 


26.1 


49.9 


1637 


45.0 


11.6 - 


772 


21. z 


An 5 

4VJ . -J 


1595 


/^43 .9 


59,5" 


2038 


56.1 


62.6 


2445 * 


67 /4 


23.3 


753 


2'0.8 


14.0. 


428 


11.8 


16/1 


463 ' 


12.7 


83.9 


3176 


. 87.3 
* 



• a Signifi£ant at the 1% level 
* includes: Hispanic, American~Indian, Alaskan Native, Asi*n and, Pacific Islander. 
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Table 4-8: Average Starting and Final Hourly Wage Rate, 
Average Weeks Stfeloyed in the Post-Program 
Period , and Percent Currently Employed by 
* Program and Demographic Characteristics 
(Number of Cases in Parentheses) 



Average Starting 
Hourly Wage Rate 



Average Final Average Weeks 
Hourly Wage Rate Employed 



Percent Currently 
Employed 



♦ ALL RESPONDENTS 


3.97 


(3087) 


4.26 


(3051) 


20.0 


(3227) 


73.2 

r 


(3588) 


% 

. • program' 


















Job Search 
.^Assistance 
Occupational 

* * « * Training 

* * AGE \ * 


3.99 
3*97 


(870) . 
(2217^ • 


4*28 
4.25 


(858) ' 
(2193) 


19.2 
20. 3/ 


(883 ) a 
(2344) 


73.6 
73.1 


(1009) 
^2579) 


16-19 
20-24 
25-44 
45-64 


3.55 
3.91 
4.20 
3.77 


(417) % 
(1010) 
(1316) 

(302) 


3.75 
4.19 
4.52 
4.03 


(414) a X 
(991) 
(1307) 
(298) 


13.9 
20.5 
20.2 
18.7 


(418) a 
(1036) 
(139a) 

(332)' 


68.2 
74.4 
74.3 
*72*2 


(465 ) b 
(1153) 
(1548) . 

(370) 



EDUCATION « 

8 or less 
9-11 

12 « * 
* 13 and^over. 



3.54 
3.75 
3.92 
4.49 



(225) e 
.(767)' 
(1425) 
(670) 



3 


77 


' (224.) a 


17.9 


(241) a 


64 


9. 


(271) a 


4 


02 


(758) \ 


18.6 


(842) 


66 


1 


(930) 


4. 


18 


(1409) , 


20.7 . 


(1462) 


75 


1 


(1625) 


4 


87 


. (660) > 


21.0 


(682) - 


80 


0 


(762) 



SEX 

Male 
Female 



4.30 (1372) 4 
3.71 , (1709) 



4.67 (1359) 
3.93 (1686) 



19.7 x (1830) C 
20.3 (1392) 



73.5 (1575) 
72.9 (2007) 



RACE 

White 
' Black 
Other 



4.02 
3.88 
3.90 



(2067) J 
'(644) 
(366) 



4.31 
4.14 
4.19 



(2046) 
(62$) 
(365) 



19.7 (2137) 
20.4 (684) 
20.4 (394) 



74.2 (2405) 
71.1 (748) 
71.1 - (422) 



AFDC FSMALfe HEAD 
HOUSEHOLD 



OF 



No 



,3.68 (356) 
4.01 (2731) 



3.89 (354) a 17.5 (419) a 
'4.31 (2697) 20.3 "(2808) 



67.8 (456) J 
",74. 0 (3132) 



1 V 



Significant at the 1% level 
^Significant at the 5% level 
Significant at the 10% level 
Hourly wage rates measured in dollars per hour.' 



9 
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The characteristics of terminees -not only influenced wages earned during the m 
follow-up period, but also appeared to be related to changes in wages 'eatned during 
that, period. Those who are older and more highly educated registered greatet gains 
in wage rates during the follow-up period than their younger and less educated 
counterparts* Moreover, males as well as non-ATDC recipients also outpaced their 
counterparts during the follow-up period, with gains of $.37 aifd $.30, compared to^ 
$.22 and $-.19, respect£vely. 

In addition to assessing the hourly wages earned by participants during the 
follow-up period, prime* sponsors may also be interested to learn about the extent 
and stability of post-program employment. As shown in Table 4-8, the average dura- 
tion of employment in the six month follow-up period was 20.0 weeks. " Moreover ,. Che 
'data reveal that many of the^expected relationships exist between weeks employed * 
and selected demographic subgroups of the respondent population. Older, more educar 
ted terminees who were not receiving AfDC tended to work more often than their 
counterparts. Unlike the findings on hourly* wages , however, terminees from OT pro- 
grams worked significantly more weeks than their J SA Counterparts . Although the 
absolute difference is small, the high level of statistical significance clearly 
suggests £hat this is not a spurious finding. 

That JSA participants worked few^r weeks ^than their OT counterparts should be 
of some concern to the prime sponsor. Since JSA participants are expected to have 
marketable skills, one would also expect them to fare at least as well in "the post- 
program period as OT individuals, whose levels of marketable skills- are the primary 
focus of the program. In vi$w of this finding, the prime sponsor may wish to take 
a closer look at ij:s JSA program in order to determine the causes of this observed 
differe^ae in performance. > 

Jtather than utilizing the number of weeks of employment to assess post-program 
employment, prime sponsors might also wish to use the proportion of participants 
currently employed as an -outcome measure. This measure is easy to^construct and is 
relatively straightforward with respect/to interpretation and usei Table 4-8 shows 
/* • -105- 
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that the expected 'relationships tend to ekist between the percent currently employed 
and the various demographic suhgroups. However, in contrast ta average weeks of 
employment, there- is no significant difference between the proportion of OT parti- 
cipants employed arid^the proportion of JSA participants employed. As described in 
Chafer Two, this type of outcomes measure dois not adequately capture the total 
posoiprogram employment of terminees And, in view of the difference between it- and 
that of weeks of employment-, clearly demonstrates the disadvantage associated with 
its use. 

^he data in Table 4-8 show two important types of summary findings.. First, 
certain types of participants benefited more than others in terms of hourly wages 
and post— program employment. This indicates a 'need to further examirfe the sources 
of these differences in order that they 'may be addressed in the future. Secondly, 
* these data show that, -on average, terminees' starting and final hourly wage-rates 
did not differ with-respect to their program of termination. It was also shown, 
however, that demographic characteristics of'OT participants differed significantly 
from those of JSA participants. S^nce participants in the JSA program tended to ^ 
» have characteristics which lead to greater labor market success (that is, oldejr, * 

mpre educated), the prime sponsor may suspect that the use of simple averages 
obscured an accurate assessment of program effects. As discussed "^arliei;, the JSA 
program is designed ,to serve a more "job ready" group of participants. As # a result, 
these average measures of program outcome may be biased upward for the JSA program 
n % - relative to the OT program. 

To address this Issue and ^provide insights into which of the two programs J work 
best for selected population subgroups, the prime sponsor may wish to adopt £n 
approach such as that shown in Table 4-9. This table presents the measures of pr^jr 
gram outcomes for the two programs controlling for differences in participant 
characteristics. As is evident from the data, this approach provides a more 
accurate appraisal of program performance. For instance, even though participant^ 
in the JSA activity are significantly older -than those in OT, wfren only the 16-19 



Kmc ' ' - 10 % 



year old age group is examined, participants in OT earn significantly higher hourly 
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Table 4-9: Post-grojratt Outcomes by Program and Demographic Characteristica 
(Kumber or Casts in^?ir enchases) , 



Prograa and 
Demographic 
Cheractsriscics 

ACE ' ' 

16-19 
Job Search Asst. 
Occupy Training- 



Average Starting 
Hourly Kage Rata 



3.28 
3.65 



(110)* 
(307) 



Average 'Final 
Hourly Kage Rate 



Average Weeks 
Saoloyed 



3.44 
3.86 



(306) 



18. 
19. 



(98) 
(320) 



Percent Currently 
EsiP loved 



68.1 
68.2 



(119) 
(346) 



20-24 

Job Search Asit. 3.74 
Occup. Training 3,97 



(258)* 
(752) 



4.01 
4.25 



(251)* 
(740) 



19.8 
20.3 



(265) c 
(771) 



73.1 
74.9 



(294) 
(859) 



Job Search Asst. 4.37 (379)* 4.71 (378) 19.8 (389) 

Occup; Training - 4.13 (937) 4.45 (929) 20.4 (1009) 



77.8 
72.9 



(442) c 
<*ip6) 



45 and over 
Job Search Asst. 
Occup. Training 



3.99 
3.66 



(101)° 
(201) 



4.25 
. 3.92 



(100) 
(198) 



16.6 
19.7 



(110)* 
(222) 



66.4 
75.2 



(128) c 
C242) 



SDUCATIOK 
Less than 8 vears 
Job Search Asst. 
Occup. Training 



3.12 
3.65 



(47)* 
(178) 



3.40 
3.87 



(47)° 
(177) 



. 9 

15.0 
18.7 



(si)*" 

(190) 



62.7 
65.6 



- (59) 
(212) 



9-11 

Job Search Asst. 3.44- (11*)* 

Occun. Training . 3:80 (654) 

* ' * 

If t - 

* Job Search Assc.. 3.70 (300) ' 

Training ~^3>-98 (1125) 



13 and ovtr 

Job Search Asst.- 4.45 (410) 
Occup. Training 4.55- (260) 



3.64 
4.09 



uur 



^16. SC (120)* 
18.9 (722) 



/ 

3.92 (295) 
4.24* (1114) 



4.82 ,(405) 
4.96 (255) 



18.8 
21.1 



*2t).8 
* 21.3 



(302) 
(1160) 



(410) 



62.4 
66.8 



70.6 
76.4 



80.7 
81.1 



(141) 
(789) 



(343)° 
(1282) 

(466). 
(296) 



SEX ' . v 

Male 

J^ob Search Asst. 4.06 
Occup. Training 4.41 

Feoale 
Job Search Assc. 
Occup. Training 

KACE . 
White 

Job Search Asst. 
. Occup. Training 

Slack ^ ' 

Job Search Assc. 3.75 
Occup. Training 3.90 



(456)* 
(916) 



3.91 (414)* 
3.65 (1295) 



4.05 (695) 
4.00 (1372) 



(117) c 
(527) 



4.32 
4.84 



<453) - 
(906) 



4.23 (405)* 
3.83 (128. 



/ 

4.34 (682) 
4.29 (1364) 



3.93 
4.19 



(H7) c 
(512) 



19.2 
20.8 




17.1 
21.2 



(463) 
(929) 



(420)* 
(1410)' 



(701) 
(1436) 



(127)* 
(557) 



73.1 
73.7 



(532) 
(1043) 



74.2 (477) 
72.5 "(1530) 



74.4 
74.1 



63.9 
72.8 



(801) 1 
(1604)* 



(144)* 
(604) 



2 j£o £ Se*rch Asst. 3.-77 (57). 4.25 (58) 20.2 (53) 

Occup. .Training 3.92 (309) , 4.18 (307) 20.5 (2£) 



85.5 
68.6 



(62)* 
(360) 



AfDC FEMALE HEAD 
OF HOPS EH OLD 
Yes 

job Search Asst. 
Occup. Training 



3.95 (33)° 
3.65 (323) 



4.08 
3.87 



(33) 
(321) 



15.1 
17.7 



(37) c 
(382) 



57.1 
68.8 



(42) v 
(414) 



2SL 

Job Search Asst. 
Occup. Training 



3.99 (837) 
4.02 (1894) 



4.29 (825) 
4.32 (1872) 



, Significant at the U level 
i Significant at the 5Z level 

Significant at che 10X level 

Hourly wage races measured in dollars per hour. 



19.4 
20.8 



(846)* 
(1962) 



74.4 
73.9 



(2165) 
(2165) 



LC 
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wage rates. Conversely, for the 25-44 /ear old and 45-and-over age groups, par- 

ticipants in the JSA activity earn sigjtif±cantly higher wa^e rages than OT partici- 

pants. Table 4-9 also shows similar results with respect to education. In this 

case, when the level of education is held .constant , it can be seen that for each v 

* 

level of education, participants in OT earn higher wage rates than participants in 
JSA programs, 

While/these findings reveal the generally superior wage producing performance 

of OT poro^?ams, there are instances when the JSA program produces higljer hourly wage 

rates. For example, for those 45 years of age and over, participation in a JSA 

program leads to higher hourly wages than participation in an OT program. Similarly, 

when the sex of participants is held constant, women from the JSA program earn higher 

hourly wages than those from OT. This contrasts with the finding for men who appear 

to benefit more from participation in OT. 

» 

Similar types of analysis are^hown in Table 4-9<with respect to weeks of post- 
program employment and the percent oJr^ffrzrmer* participants employed at follow-up con- 
J^:t. While little variation was seen with respect to the latter outcomes measure, 
significant differences did emerge between participants' level of education and 
weeks of post-program employment. For each level of >*edu£at'ion, OT participants 
worked significantly more weeks during the follow-up period than their JSA counter- 
parts* 

. The above analysis clearly shows the usefulness of* follow-up data in reviewing* 
post-program performance. At the same time, however, the analysis shows the short- 
comings of simple comparisons. When the distribution of participant characteristics 
is controlled for, otherwise unobserved differences in program performance become 
evident. While this is a useful approach, it is also quite cumbersome. Moreover, 
it does not control for additional potential differences in characteristics, such as 
those between young black women and young white won\en. Although additional tables 
can be constructed to accomodate evermore detailed analysis, this *mSy result in an ; 
analysis that is unwieldy and not readily inter pretable. A prime sponsor with 
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access to automated data processing facilities may wish to consider using multiple 
regression analysis for a more precise measure of program outcomes. A brief descrip- 
tion follows for those who may be interested* ' 1^ 

< 

Dr. Multiple Regression Analysis of Program Performance ' * '* 1 

The analysis and results of Table 4r9 have two major shortcomings. First, 
they do not make clear what the magnitude of the difference is in the performance of 
OT and JSA programs. While it may appear that OT is more effective than JSA, 
differences between zero and foi/^y cents per hour for wage rates, and zero to three 
weeks for weeks of employment measures ajtre observed. Second, the table shows results 
whi#cti hold constant only one variable at a time. The drawback to this approach is 

0 

that all of the personal characteristics vary/simultaneously across the two programs. 

To address these shortcomings, the prime sponsor may wish to use multiple regression 

* • 

analysis. To illustrate the use of this technique, the following regression equation 
is estimated: * 



W - a A + a, AGE1 + a„ AGE2 + a 0 AGE3 + a. ED + a. SEX + 
0 1 2,3 4 5 

a. RACE + a_, WEEKS EMPLOYED + a Q PROG 
o 7 o * 



This equation assumes that the hourly wage of former participants (W) , is 
determined by their age, education, sex, race, weeks of employment in the six months 
prior to CETA, and program of participation. The variables which have been included 
in the equation have the following meanings: 

o ^ The age of the participant is included by the use of three, vari- 



ables :• 



r 



- AGE1 has a value ofotte if the , participant is between 
16-19, inclusivej/and z v° otherwise. 

- AGE2 has a value of one if the , participant is between 
20-24, inclusive, and zero otherwise. 

- AGE3 has a value of one if the participant is 45 or 
over, and zero otherwise. , , 
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The estimates'of a^, ^ and a3 therefore, measure the 
* difference between the wage rates earned by participants 

in the three age groups listed here and participants in 
' the 25-44 age group, the base group for comparison. 

Since individuals in the prime age group, 25-44, tend to 
have higher earnings than all others, the estimates of 
a l» a 2 atK * *3 should be negative. 

o The variable ED represents the years of ,educatioc com- ^ 
pleted by the participants. The coefficient a4 measures 
% the number of cents per hour more an individual earns on 
average for an additional year of education. 

o SEX is a variable which has a value of one if the in- 
dividual is a man and zero if a woman. The estimate of 
as, therefore, measures how much more per hour men earn 
than women. 

o f RACE is a variable which has a value of one if the in- 
dividual is white, and zero otherwise. The estimate of # 
ag measures how much more per hour whites earn over all 
.'others. K 

o WEEKS EMPLOYED measures the total number of weeks partici- 
, pants were employed^in the six-month 'peripd prior to the 

program. The estimate of ay measures .how much More per 
hour is earned for 'each additional week worked in the six 
, months prior to the program. 

o The variable PROG has a value of one if the individual ♦ 
particicipated in OT and zero for JSA participants. The 
coefficient ag measures the impact of participation in OT 
relative to JSA.. It is the estimate of ag which is of 
most interest. 

J 

Table 4-10 presents the regression results with respect to the hourly wage 
earned on the first post-program job. The first column shoys the estimate of the 

equation specified above. From this estimate, everything else equal, the follow- 

i 

ing result emerge: 

o Individuals aged 16-19 earn 57 cents less per hour than in- 
dividuals aged 25-44. Individuals aged 20-24 earn 41 cents 
less per hour and individuals aged 45 and over earn 24 cents 
less per hour than this same base group (25-44); 

* o An additional year of education yields individuals an aver- 
age wa&e advantage of 15 cents per hour; 



o Males earn 49 cents per hour more than females; 

o White participants earn 14 cents per hour more than all other 
participants; I . 

o An additional week of employment in the year prior to the 
program leads to 0.3 cents per hour more; and 



ERLC 



Participants in an OT program e&rn 27 cents per hour more 
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relative to those in a JSA program. 
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Table 4-1Q: Regression Analysis of Starting Hourly Wage Race 



/ 


(1> 


(2) 


(3) 


(4) 


Age: 16*19 


-57.0* 
(9.63) 


-55.5* 
(9.63) 


•-36.2* 
• (9.61) 


-56.3* 
(9.61) 


Axe: 20-24 ^ 


-41. 1** 
(6.82) 


-40.9* 
• (6.31) 


-40.8* 
(6.79) 


-40.5* 
(6.30) 


Age: 45-f „ 


-24.3* 

^9:86) ' , 


a 

-23.8 
(9.84) 


-24 . 2* 

(9.82) 


-24.0* 
(9.32) 


Ta«t« o£ Riueation romlt t #<1 


14.8* 

(1.35) 


14.6* 
.(1.35) 


14.8* 

(1.35) 


14*8* 
(1.35) 


Sex * 

M 


49.5* 
(5.98) 


,49.9* 

(6.00) 


51.6* 

(6.00) 


51.3* 

(6.01) * 


Race^ 


13. 7 b 
(7.48) 


13. 7 b 
(7.47) 


15.3 b 
(7.47) 


15. 6° 
(7 .47) 


Weeks Eap loved In Tear Prior 
co Prog ran 


0.3b 
(0.18) 


0.3 b 
(0. 18) 


0.3° 
(0.18) 


0.3° 
(0.18) 


Cons cane 


183.7* 

(19.47) 


202.7* 
(20.95) 


171.2* 
(23.0) 


167.3* 
(23.8) 


Occupational "Training 


27.2* 

(6.60) 


( 13.8 b 
(7.92) 


19.7? 
(8.11) 


19.7* 
(8.15) 


Length of Training: 12 or 
Fev«r Week* 




-20.2* 

(8.14) ■ 


-17.4° ^ 
(8.16)/^ 


^7^4° 

(8.16K 


Length of Training: 24-36 
Weeks 




0.5 
(9.58) - 


(9.56^ 


-1.4 
(9.59) 


Length of Training: Mora 
Than 36 Weeks 




9.01 


6.15 


6.22 



(U.43) (11.43) (11.44) 

Completed - - 27,6* 

(8.47) 

Not romp leted and Placed - - - 10 *' 1 _ 

* (15.22) % **V* 

b • ' 

Completed and Moc Placed - - - 23.0 ** • 

(13.67) 



Completed and Placed 



0.13 0.14 0.14 

# 26.3* 25.1* 21.7* 

U/ 1803 12 / 1802 14 /1 800 

(S-1815) 

*Signif leant at the IX lavel 
b Slgnlf leant at the 52 level 

Significant at the 102 level ' 
Standard errors df coefficient estimaces in the parenthesis. Unit of measurement 
ia cents per hour. 



S? 0.13 
F 34.8* 
Degrees of Freedom 8/1806 
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Baaed uptfi^/these findings, it would appear that the independent effect of 
being a participant in an OT program is 27 cents per hour. However,' since OH is 

' • 1 

*a much longer activity than JSA, it, is not clear whether the relitivej.y positive 
effect of the program is due to the length of training per se, or^tc- an inherent 
advantage of the OT program. To examine this issue, an additional variable has 
been ad<Led to the equation. This variable measures the number of freeks each in- 

4 * „ * 

dividual 'participated in a program in the following manner: 

o * Length of Training: 12 or Fewer Weeks has a value of one if the 
participant was in the activity for 12 or fewer weeks, and 
zero, otherwise. - * t % 

o Length of Training: 24-36 Weeks has a value of one if the par- 
ticipant was in the activity for 24-36 weeks, and zero other- « 
wise. ' 

o Length of Traihing: More than 36 Weeks has' a value of one if 
the participant was in th£ activity for more^than 36 weeks,/ 
- and zero otherwise. ,i 

The estimate of the coefficient on each of these terms measures the difference 
between the wage rates earned by participants who are in one of these tftree^groups 
and those wttq participated for between 13 and 23 weeks as the base group' for com-' 
parison, The results of adding the variables to the first equation are pres>nt&d 
in column two of Table 4-10. * 

As can be seen, the e^fec^ts of participant characteristics are nearly identical 
to those obtained from the first estimation. However, the estimate of the effect of 
the OT relative to the JSA program has declined substantially to 15 cents per hoyr. 
The reason for this decline is that participants in OT were in the program for a 
much longer period of time thati participants in the JSA program. Thus, without the 

length of training variables, the program variable captures thfc effect of both the 

/ 

number of ' weeks a participant was in an activity and the effect of being in OT 
relative to JSA. When these variables are entered separately, everything else eqtial, 
participants in OT earn 20 cents less per hour 'in comparison to those who partici- 
pated for 13 to 23 weeks. For those participants who were in an* activity for 24 or 
more weeks, additional weeks of program participation did not add to their starting 
hourly wage rate relative to those who participated for 13 to 23 weeks. 
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Although *there appears to be a positive relationship between earnings and 
duration of training, the fact that increasing levels of program participation 
beyond 23 weeks did not add to earnings should be, of* some concern to prime 
sponsors. It could be that the linger training psogr^ms were not well designed 
or thafr the longer programs offered training in the* slower growing occupations. 
Alternatively, since the duration variable reflects a composite of all training 
efforts, it Qould be capturing the effects of either one or two longer programs, 
or certain individuals who remained in a program the longest but did not perform 
well. Prior to making final judgements of this finding, prime sponsors should, 
therefore, examine the reasons^or its occurrence if well directed efforts at 
program improvement are to be accomplished. * 

In addition to considering the duration of program participation, the prime 
sponsor will undoubtedly be aware^ that those participants in OT who$e lepgth of 
training" is brief are most likely thos'e who dropped out and did not complete the 
activity. Since these participants may alscT earn significantly lower starting 
hourly wage rates, it is necessary to control for participants' completion status. 
To accomplish this, an additional variable was added to the equation specif ied 
above. The variable has a value of one for an individual who completed the 
activity and-^a zero otherwise. When the '-equation is re-estimated^, column 3 of 
Table 4-10 shows that, holding all else constant, program completion is important 
and leads to a 28 cents per hour wage advantage. Also, controlling for their com- 
pletion status, participants in OT earn 20 cents per hour more than participants 
in JSA after controlling for personal characteristics and the length of the training. 

The prime sponsor may also be interested in the potential influence of job 
placement on participants' starting hourly wage rates. - 'Immediate placement in an 

unsubsidized job holds a unique position in the evaluation of prime sponsor programs. 

t 

, On the one hand, a placement is an outcome of an activity. When viewed from such a 
perspective, variations in the placement variable would be explained by many of the 
demographic; and program characteristics which have been employed thus far. "On the 
J -113- 
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other hand, placement is also a variable which cln be used to explain why some par- 
ticipants earn a higher starting hourly wage rate and others do not. It is this 
latter approach which is used here* 

To examine, the potential impact of placement and still control for completion 
status, four categories Were created. These categories cover participants who: 

o Were not placed aftd 'did not complete; 

o Were placed and did not complete; < 
o Were not placed and completed; and 
p Were placed and completed. 

Since individuals in each of the latter three categories experienced more posi- 
tive i immediate program outcomes, it is anticipated that they will fare better than 
the base group (those who did not complete and were not placed) , Column 4 of TAble 
4-10 shows that the effects of personal characteristics on the first hourly wage 
remain similar in magnitude to those of previous equation estimates. The effect of 
participation in OT relative to JSA is 20 cents per hour, also similar to that 'in " . 
the'previous equation estimate. For individuals who participated 12 or fewer weeks, 
as in the previous equation, 18 cents less per hour is earned relative to those who 
participated for 13 to 23 weeks, ♦ J 

• Estimates of the coefficients on each of the three 4 newly created variables^are, 
as expected, positive. However, the'estimate of the coefficient on the variable^ for, 
those who were placed but did not complete is not significantly different from zero,, 
This shows that those who did not complete but were placed do not earti significantly 
. higher starting hourly wage rates than those who did not complete and were not 
placed.* For individuals who completed the program, whether or 'not they were placed, 
higher starting hourly wage rate*5 are learned. These findings, therefore, indicate 
that completion of' an activity may very well be a mo're important determinant of the 
hourly wage rate on the first post-CETA job than immediate job placement. 
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The results shown thus far have concentrated on the starting hourly wage rate 
of the first" post-CETA job.-i The prime sponsor may also wish to estimate the same 
set of equations for the wage earned on* the last or current post-C^TA job. Results 
for the estimation of such an equation are presented fin Table 4-11. As can be seen, 
estimates of the coefficients of participant characteristics are very similar to 
those shown fn the previous table. The actual values of the coefficients are, how- 
ever, ~ somewhat larger, due principally to the fact that the average final hourly 
wage ratej^s $4.26, while the average starting hourly wage rate is $3.97. For the 
program variables, the sam2 basic patterns are shown in Table 4-11 as were shown in 
the previous analysis* Th,ere are, hdwever, two important differences. First, the 
effect of the OT program relative to the 1 JSA pr£granLi&_uow only 13 cents per hour 
as opposed to 20 cents per hour for the hourly wage rate on the first post-CETA jol\. ' 
This represents approximately a' 35 percent reduction in program effectiveness over a 
six-month period. The second difference from the previoui results is the appearance 
of a more important role for immediate job placement. Relative to those who were 
not placed and did not complete, the findings show that those who were placed and 
did'not complete as well as those who were placed and Completed earned 39 and 41 
cents per hour more, respectively. Tho^fc-vWho Completed but were nctt placed earnei 
23 cents per hour, more. Thus, both c<^letionkn4 placement play an important role. 
However, placement is somewhat more important as shown t by the 41 cents per hour 
more earned by those who were placed and completed, relative xo the 23 cents per 
hour more earned by those who were not placed but completed. Over time, one f s 
placement status becomes more important in influencing ' wages r while at the, time of 
termination, completion is the more influential fyf ,t:he two. 

„ - In addition to analyzing 'hourly wage rates, prime spofcors may also wi$h to 
use regression analysis to examine weeks of employmejafi^in the post-program period. 
An equation similar to that presented above is, in Table 4-12 for the total weeks 

o£ post-program employment. This equation, hcjwever, includes only those terminees 

: * t 
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Table 4-11: Recession Analysis of Final Hourly gage gate 





(1) 


f 7\ 

hZ; 






Asa:. 16-19 ) , < 


-65. 0 a 
( h.jj; 


-63.3* 

111.?**/* 


-63,3* 

Ml *iL\ 
i 


'-63.8* 


A«a: 20-24 ' 

• 


r44.3* 
(8,21) 


-44.1* 
(8.19) 


-44.0* 
. (8.15) 


-42.3*. 
(o.xo; 

* 


A«a: 45* 


-25.1 b 
(11.85). 


(UXa3) 


-25. 2 b 
(11.32) 


-24. 7 b 
(ll v *80) 


Tears of Education Coepleced 


16^ 
(1.62) 


16.0* 
(1.61) 


16.1* 
(1.62) 


16.0* 
VI. ol; 


Sex . 


59.8* 

f 7 10\ 

(7.19) 


60.4* 
w • Ay ; 


61.6* 


61.1* • ^ 


Race 


14.3 C 
(9.0J.) 


, 14 .5 C 

( Q QQ\ 

(8.99; 


15, 8 b 
V».U1; 


16. 2 b 

f o nfu 


Weeks Employed In Tear Prior 
co Progrim 


(0.21) 


0.4 b 
(0. 21) 


/o 7i ^ ^ 


\\j . ii/ 


Cons cane 


186.9* 
(23.3) 


215.0* 
f (25.1) 


192.6* 

f 77 £\ 

^z/ .o; 


176.7* 

^ £o • ** ; » 


Occupational Tralnins 


27.7* 
(7.95\ 


^ 9.4 
(9.51) 


13.6 C 
(9.74) 


'12.5* 


Lenfth of Tralnins: 12 or 
^ Fever Weeks j 


f 


%- -29.9* , 
(9.75) 


-27.9* 
(9.79) 


-28.3* ' 
> (9./o; 


Lensch of Tralnins: 24-36 
- Weeks 




-3.8 
(11.53) 

<* 


-4.5 
(11.52) 


-6.3 ' 
XU.54) 


Lensth of Tralnins: More Chan 
36 Utek* 




8.2 
f (13*68) 


' 6.1 

(13,72) p 


6.0 

(1 7 7fi\ 

lu./u; 


Couple ted 






19. 8 b 
(10.1) 




Hoc Coopleced and Placed 


- 


- 




39. 3 b 
(18.09) 


Completed and Hoc Placed 








22.8° 
(16.4) 


> Coapleced and Placed 

i 


0.12 


0.12 


* a* 

• o.u 


41.4* 
(13.40) 

' J 

0.13 


7 


31.09* 


' 23.90* 


22.26* 


19.68* 


Desrees of Frtedo* 


8/1783 


11/1780 


12/1779 


14/1777 


(N • 1792) * 










, *Sisnif leant ac che IX l«vel . 
b Sisnlf lctnc ac che 5X level * 
' c 31snlf leant ac che 10X level 

Standard Errors of *co efficient tstlaates In che parenthesis. Unic of measurement 
Is canes per hour* | •* 

*A one uU ctsc of the hypothesis 'thac che Coefficient is equal to zero, £s 
rejected ac the l'0.05X level* 






-116- 




> 






126 


s 


* 



- ♦ 



\ 



« 



Table 4-12: Ratrcaaion Analysis of Total Weeks Worked 





(1) 


<2> 


(3) 


(4) 


. 


age: 16-19 


-L29 

(i.ax) 


-1.09 
/1. 80) 


-1.14 
(1.30) 


'-1.18 
(1.30) 


-1.10 
(1.77) 


Age: 20-24 


(1.14) 


-0.42 
(1.13J 


-0.28 
(1.13) 


-v -0.36 
0 * (1.13) 


-0.19 
(1.11) 


• Age: 45^* 


•-0.36 
(0.42) 


-0.31 
(0.42) 


-0.29 
(0.42) 


-0.28 
(0.42) 


-0.29 
(0.41) 


• Ttara of Education Completed 


0.31* 
(0.08) 


0.21* 
(0.08) 


0.21* 
(0.08) 


0.22* 
(0.08) 


0.21* 
(0.08) ' 


Sex 


0*14 
(0.35) 


^0. 16 
(0.35) 


.-0.20 
(0.36) 


-0.13 
(0.36) 


c -a.20 

(0.35) 


Uca 

■» 


Q. 13 
(0.42) 


0.02 
(0.42) 


0.04 
(0.42) 


0.10 
(0.42) 


0.17 
(0.41) 


Weeks JEaployed In Tear Prior 
co Program 


0.03* 

(0.01) m 


*0.03* 
(0.01) 


0.03* 
(0.01) 


0.03* 

(0.01) 


0.02 5 
(0.01) 


Constant 


16.62* 
(1.12) 


15.60* 
(1.13) 


16. 33*t 
(1.22/ 


15.14* 
(1.35) 


' 13.4V 
(1.37) 


Occupation*! Training 


2.05* 

(0.38) 


C91* 
• (0.38) 


1.42* 
(0.45) 


/-1.64* 
, (0.47) 


1.53* 

< (0.46) 


tfuaber of Dependents 


0.017 
(0.13) 


-0.06 
(0.15) ' 


-0.05 
(0.15) 


-0.04 
(0.15) 


-0.04 
(0.14) 


Aid co FiaHlti vlch 
Dependent Children 


-1.20* 
(0.51) 


-1.07 b 
' (0.51). 


-1.07 b 
(0.51) 


-1.06 b 
" (0.51) 


(0.30) 


Starting Hourly Hage Race 

/nv Pa4P— ££TJL .fob 


- 


0.006* 
(0.001) 


0.006* 

(0.001) 


0.006* 
(0.001) 


0.006* 
(0.001) 


Lengtk of Training: 12 Or 
Fever Week* 


- 


. -\ 

t 


-0.70° 
(0.46) 


-0.59 c 
(0.46) 


-0.66 c 
(0.45) 


Length of Training: 24-36 
Weeks 


- 




0.47 
(0.54) 


0.45 
(0.54) 


0.16 
(0.53) 


Length of Training: Mora 


- 


- 


-0.31 
(0.64) 


-0.41 
(0.64) 


-0.45 
(0.63) 


Coatplacad v 








1.00° 
(0.49) 


_ * 


Hoc Complttad and Placed 


- 




— 




4.13* 
>(0.36) 


Completed and' Sot Placed 


- 


- 


- 


V 


0.19 
' (0.78) 


Computed and Placed t 




• 






3.53* 

(0.64) 


I 2 


0.027 


0.04 , 


0.04 


0.04 


0.08 , 


r 


5.427* 


7.16* 


5.99* 


5.88* 


- §.14* 


Degrees of Frtedoat 


10/1591 


11/1590 


14/1587 


15/1386 


17/1584 


(N-1602) . 













•Significant ac che IX level 
b Slgnif icanc ac the 5Z level 



'SlgtUficant ac the 10Z level 
Standard errors of coefficient astlswtes in the perenthasis. Unit of measuremanc 

weeks. 



who worked after the program. ^ 

Many of the variables included in this equation are similar to those used 
earlier. One new variable which was introduced, however, is the hourly wage rate 
on the first post-CETA job. This variable As an important element of this equa- A 
tion because it is generally thought that as participants' wages increase, ^every- 
thing ^-se equal, they will be willing to work more. The number of dependents is 
also included" because even if two individuals receive the same wage rate, the in- 
dividual with the larger number of dependents may tend to work less^due to home 
care responsibilities. Similarly, individuals eligible to receive AFDC may also 
tend to work less, due to the incentive structure of the program, even if they 
receive the same wage rate as those not eligible for AFDC. 

Table 4-12 shows "the results.of this series of equations. The first version 
of the equation is. presented* in column one. The findings show that as education 
and weeksyemployed in^the year before the program increase, weeks employed in the 
post^j>rlSgram period increase as well. The estimate also shows that, as hypothesized, 
AFDC recipients work less thaii non-recipients. Also, participants in the OT program 
work, op/average, two weeks longer than participants £n the JSA program. In the 



second column of the tab^4> the . s tar ting hourly wage-rate is added as an independent 
variable. The only major difference in the second equation from the first is that 
the magnitude of the coefficient estimates has been reduced. 

Having identified an initial version of the weeks of employment equation in 
column 2 which adequately explains the employment experience of the participants, 

i 

the prime sponsor may wish to study further effects of the program by adding the' 
length of training and the completion status of participants • As shown in column 

/ V 

I ■ I 

3 of Table 4-12, individuals who participate in activities for brief periods tend 
to work less, on average, than those who participate for 13 to 23 weeks. 

As was indicated in the discussion of the hourly wage rate equations, however, 
these results may be somewhat misleading in that some^of those who participated for 
the briefest time period may not have completed the program and, thus, may not have 
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obtained a full set of services. Therefore, the estimate in doltimn 3 that par- 
ticipants in OT work 1.4 weeks longer than participants in JSA may be too low. To 
remedy this problem, column 4 presents an equation which includes the completion 
variable. It shows that participants who complete an activity work one week longer 
than those who do not complete. Also, this version of the equation shows that 
participants in the OT program work one and two-thirds weeks longer than partici- 
pants in the JSA program, greater than that observed in fhe third equation. 

Finally, the prime sponsor may want to study the effect that placement has on 
employment. To do this, the variables which measure completion and' placement status 
That were employed in the wage rate equations are used here also. Column 5 shows 
the estimate of the equation with these variables included. There are two noteworthy 
results. First, those individuals who completed and were placed and those who did^ 
not complete but were placed worked significantly more weeks than those who were not 
placed, independent of whether they completed or not. This shows the importance of 
placement in positively influencing post-program employment, and is in contrast to 
the findings on the first post-program wage rate. Second, participants in OT tend 
to work one and .a half weeks more than participants *ln JSA programs. 

These findings have clear implications for the prime sponsor. FiMt, the 
findings show the usefulness of multiple regression analysis in isolating the effects 
9f participation in a program. Secondly, they reveal that, independent of the 
characteristics of those reviewed, occupational training programs are significantly 
more effective than job search assistance programs. t While this particular finding 
may not be relevant to any one sponsor, it does clearly show how follow-up data can 
assist in making program mix decisions. Third, the findings show that, regardless 
of the program, certain groups of the enrollee population will not benefit as much 
as others. This is the case, for instance, with 1 women, AFDC recipients, and those 
with limited prior work histories. With these findings, prime sponsors would be 
well equipped with valuable information to use as the tarsttS for further exploring' 
the issue of what, works best for whom, andVhy. Finally, these findings indicate 
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the need to carefully specify all aspects ot fcrogram participation since the dura- 
tion of participation as well as the completj^^and/or placement status has an^ 
important influence on each of the outcomes measures* It is important not to treat 
program participation in an aggregated fashion, but to provide information on 
specific areas of programs that may warrant special attention. 

E. Skill Training Areas and Measures of Program Outcome 

Tne results which have been presented thus far have been of an aggregate or a 
summary 'nature. While they are useful for determining where, and in many cases, how 
programs can be improved, the prime sponsor may also want a more detailed examina- 
tion of why program outcomes dif f e r^fr€tween OT and JSA participants* Table 4-13 
represents one way to address this issue, showing each of the four outcome measures 
by individual skill training area. As can be seen, substantial differences exist 
between the individual training programs. Among skill training areas with higher 
hour/y^ wage rates and weeks of employment are computer, machine > metal and elect w 
ronics related training, all of which have higher hourly wages and levels of employ^ 
ment than those produced by the JSA programs. These results partly explain 
the Source of the post-program wage advantage experienced by OT participants . The 
results also clearly show the importance of the occupational selection process in 
influencing post-program performance. v With this information, prime sponsors would 
be in a better position to take those steps necessary to improve local performance. 

F. Participants Not Employed at the Time of the Interview * | 
An additional way to gain further insights into the results obtained thus* far 

is to examine the qualitative aspects of terminees' post-program labor market 
experiences. For example, although earlier findings revealed that 73 percent of 
the terminees were employed at follow-up contact, infQrmation on the status of the 
remaining 27. percent can be useful for identifying th^e aspects of program design 
requiring improvements. Table 4-14 shows that approximately 60 percent of those 
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Table 4-13 Starting and Final Hourly Wage Rate by 

SkiJJ. 'Training Area for Occupational Training (N=2196) 



Training Activity 
Total 

Clerical Related 
Electronics - 
Health Related 
Metal Related 
Auto and Engine 
Food 
Machine 
Computer 
Printing 
Maintenance 
Carpentry 
Accounting 
Other 



Starting Hourly Final Hourly 



wage Rate c 

3.94 „ 
^ 3.56 

4.20 

3.75 

4.72 

J. 63 

3.44 

4.66 

5.57 

3.60 

4.50 

4.13 

3.35 

3.94 



Wage Rate * 
-4.21 
3-76 
4.54 
3.90 
5.12 
3.88 
3.61 
5.28 
5.87 
3.83 
5.06 
4.11 
3.41 
4.17 



Weeks 
Employed* 

20.2 

20.7 

~21.8 

19.5 

22.0 

21.4 

17.0 

23,0 

23.7 

21.2 

20*.7 

16.2 . 

16.0 

18.6 



Percent^ 

Currently 

Employed 5 

72.0 

75^J0 

77.5 

67.4 

63.6 

66.7 

58.5 

83.5 

83.3 

78.8 4 

68.4 

64.8 

52.2 

67.9 



Percent 
100.0. - 

29.7 

16.0* 
5.3 
3.1 
3.9 
3.0 

10.9 
2.4 
2 .7 
1.5 
2.0 
3.2 

22.4 



Significant at the 1% level 

Hourly wage rates measured in dollars per hour. 
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fable 4<L4: Participants Currently Looking for Work and 
Methods Used to Seek Work and Participants 
Not Currently Looking for Work and Reasons 
for no Job Search 



Occupational Training 
Number Percent 



Job Search Assistance 



Number 



Total 



Percent Number Percent 



All Respondent^ 



626 



71.5 



2 50 



28.5 



876 * 1Q0.0 



Currently Looking for 
Work 4 



358 



i 

57.2 



157 



62.8 



515 



58.8 



Currently Not Looking 
for Work 



268 



42.8 



93 



37.2 



361 



41.2 



Methods Used to Seek Work 



Training Agency 
DES; 

Private Agency 

Friend 

Newspaper 

Walk-in 

School 

Other 



15 


4.2 


7 




4.6 


22 


4.3 


27 


7.6 


13 


0 


8.5 


40 


7.8 


13 


3.6 


8 




5.2 


21 


4.1 


20 


5.6 


4 






24 


4.7 


216 


60.5 


87 




56.9 


303 


59.4 


50 


14.0 


17 




•11.1 


67 


13.1 


1 


.3 


* 3 




2.0 


4 


.8 


15 


4.3 


14 




9.3 


29 


5.7 



Reasons for Not Lop king 
for Employment 

Inadequate Training/Skills 
Unsure of Myself 
Personal Problems/Health 
Child Care/Pregnancy 
Tired of Looking 
Other 



7 


2.7 


3 


3.3 


10 


2.8 


3 


1.1 








.8 


48 


18.3 


16 




64 


18.0 


83 


31.7 


6 


6.5 


89 


25.0 


7 


2.7 


4 


4.3 


11 


3.1 


114 


43.5 


65 


70.7 


179 


50.3 
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/ 

who were not employed at follov-up contact were looking for work at the time of 
the interview. The table also shows that of those individuals who were looking 
for work, 60 percent of the 0T participants and 57 percent of^"the JSA participants 
were using the newspaper as- the major source for finding work. In addition, 14 
percent of the 0T participants and 11 percent of the JSA participants were simply 
walking into firms to identify job openings. While these methods may be an effec- 
tive approach for some, more sophisticated methods could be used, especially by 
JSA participants who had recently received training in job seeking techniques. 
While such a finding is not necessarily critical of the JSA program, it should 
cause the prime sponsor to carefully examine the program* Such an examination 
should include a careful review of the types of jot seeking techniques participants 
are being taught, 

Finally, Table. 4-14 also shows several reasons why non-employed participants 
are* not looking for work. Child care/pregnancy is one of the reasons which appears 
frequently. This result should be viewed in* conjunction with the demographic charac- 
teristics shown in Table 4-15. This table shows that the distinguishing demographic 

characteristic between those /looking for work and those not looking tor work is that 

/ 

nearly 70 percent of those no^ looking are female, whereas only 46 percent of those 

1 

looking are female. Therefc/re, the prime sponsor may want to take a more careful 
• look at the availability of child care facilities. It may be the case that many 
women find it necessary to ieave the labor markej: due' to the lack of satisfactory 
child care arrangements. ^ 

-This analysis of follow-up data has provided several illustrations of how 
follow-up data^can be used in a simple and then more sophisticated manner to ^assess 
program performance. Although it -does not reflect all the possible* ways that 
follow-up data can be used for local planning purposes, it does provide a -first step k 
for those prime sponsors interested in examining the pos^program performance of 
their local employment! and training programs. 
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Table 4-15: Distribution of Participants Not Currently Working 
* By Demographic Characteristics 



ALL RESPONDENTS 



Looking 
Number Percent 



515 



58.8 



Not Looking 
Number Percent 



361 



41.2 



AGE 

16-19 ' - 
20-24 
' 25-44 

45 and over 
i 

f EDtoCATION * 

• 8 and less 
9-11 
12 

13 and over 
SEX* 



Male 
Female 



82 
153 
211 
, 66 



57 
171 
211 

76 



277 
238 



15.3 
29.1 
40.5 
12.2 



.11.1 
33.2 
41.0 
14.8 



53.8 
46.2 



58 
113 
147 

41 



28 
116 
157 

60 



109 
252 



15.5 
30.7 
40.1 
10.8 



7.8 
3$.l 
43.5 
16.6 



30.2 
69.8 



RACE 

White 
; Black 
« Other , 

AFD.C 

Yes 
No 



347 
112 
55 



58 
457 



67.5 
21.8 
10.7 



11.3 
88.7 



217 
95 
48 



71 
290 



60.3 
26.4 
13.3 



19.7' 
80.3 



^gnificant at the 1% level 
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FOOTNOTES TO CHAPTER FOUR 



1* Officially, "the Department of Labor calculates the rate of job placement as 
follows*: . 

\ 

Number of Job Placements X 100 

'Total Terminations - Intertitle Transfers 

2. This does not mean, however, that placement into unsubsidized employment 
will lead to gains in long-term employment and earnings. The degree to 
which job placement is a reliable predictor of gains in post-program earn- 
ings is yet an unresolved empirical issue. For a review of existing 

' evidence see: 

, Bonis, Michael, "Indicators qf CETA Performance", Industrial and Labor Rela- 
tions Review , Volume 32, No. 1, October, 1978, pp. 3-14. 
i 

Gay, Robert, Validating Performance Indicators for Employment and Training, 
Programs , U.S. Department of Labor, Employment and Training Administration, 
Office of Research and Development, Washington, D.C., April, 1978. 

Geraci, Vincent, and Christopher King, Employment and Training (CETa) Program 
Performance: Long-Term Earnings Effects and Short-Term Indicators , University 
of Texas at Austin, Department of •Economics, December, 1980. 

3. 4 For a more complete discussion of the issues related to the use of pre- and 

post-comparisons, see Chapter Two, Section II. A. 

4. Tests between means (simple averages), are a useful statistical technique 
for ou t come s, expressed in continuous terms, such as hourly wage rates, weeks 

* ' worEed or earned income. Such outcome measures can be compared across any 
number of program strategies or population subgroup categories* A second 
' % .technique which may be useful is contingency table analysis. This approach 
relies 1 upon the cross-tabulation of two variables, typically a "program out- . 
come and a particular demographic or programmatic subgroup of the population, 
and allows prime sponsors, through the use of a "chi-square" statistic, to 
determine if a statistically significant relationship exists between the two 
variables. ' *■ ' ft . * 

5. For a review of the nature of job search assistance programs see: 

Wegman, Robert G.,."Jdb Search Assistance: A RevjJLew 11 , Joiftnal of Employment 
Counseling , Volume 16, No. 4, December, 1979. 

6. For 23.4 percent, the area in which training was received was unidentifiable. 
For about ?5 percent of these participants, the area of training was not 
available from the Prime Sponsor MIS. For the remaining 25* percent, enroll- 
ment was in a particular training in wjiicji only a limited number of in- j 
dividuals* participated. r 

7. This effort recently became part of a region-wide effort to develop the pro- 
gram evaluation capabilities of prime sponsors. The work has Received joint 
funding from t^je Massachusetts State .Employment and Training Council (SETC), 
individual' prime sponsors, and the ETA, U.S. D.O.L. For complete documenta- 
tion see: 
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Zornitsky, Jeffrey, Glen Schneider and John Dorrer; The Development and Use 
of Local Evaluation Systems: An Employment and Training Perspective (Pre- 
pared for: Office of- Policy, Evaluation ajid Research, fitoployment and Train- 
ing Administration, U,S. Department of Lara:; Washington, D.C., September, 

i*8oo . „ % ; • j 



j 
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Technical Appendices to Part II 



W-2 Wage and Tax Statement . . 

An Empirical Comparison of "Alternative Survey Methodologies: . 
^Results of Employment, and Training Administration Field StudV 

Telephone/Personal. Visit Follow-Up Questionnaire 

Mail Survey Follow-Up Questionnaire 

Regdon^One Follow-Up 'Evaluation Project Questionnaire 

Participant Information Form 

Sample Introductory Letters and Business Reply Postcards 
A Review of Sampling Procedures 

A Review" of Costs of Operating Local Prime Sponsor Follow-Up 
Systems 




r 
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APPENDIX A 
Vf-2 Wage and Tax Statement, Copy A 
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An Empirical Comparison of Alternative Survey Methodologies: 
Results of Employment and Training Administration Field Study 
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AN EMPIRICAL COMPARISON OF ALTERNATIVE SURVEY METHODOLOGIES: 
RESULTS OF EMPLOYMENT AND TRAINING ADMINISTRATION FIELD STUDY 




Introduction 

For some' years now, prime sponsor follow-up systems, as have been described in 
this Technical Assistance Guide, have been operating in many of the prime sponsors 
in Region I. As fiscal year 1982 begins, twenty-four of the twenty-five prime 
sapors in the region are engaged in participant follow-up and their joint efforts 
constitute a regional evaluation system. The follow-up projects operated :by each 
of the twenty-four p<rime sponsors generally employ the telephone interview method 
as a way of locating, contacting and interviewing participants. In fiscal ypar 
1979, available data reveal that the telephone interview method has led to 57 p,er- 
cent of the terminees being contacted and 51 percent completing interviews. An 
interest arises out of this experience regarding'' the use and potential advantages 
of alternative follow-up survey methods, particularly in terms of the number of 
terminees who would be contacted and % interviewed, the quality of the resulting 
data and the overall .survey costs. In response to these issues, the Employment 
and Training. Administration, U.S. D.O.L., recently funded a survey research study 
in Regions VI and VII to test alternative survey methods. The purpose of this 
appendix is to report on this study. 

This appendix is composed of three sections. Section I is a discussion of the 

study's survey design. This design required that participants be randomly divided 

into three groups, each of which would be located and interviewed by one of three 

survey methods;' telephone, personal visit or mail. ^Section II details the specific 

contact, location and interview strategies used in each of these three methods. 

i 

Section III presents the findings of the study. These results are divided into five 
areas: 1) a review of the demographic characteristics of those participants in- 
cluded in the study; 2) a discussion of the jagnitvide of participant contact and 
interview completion rates; 3) a study of the determinants of the contact and com- 
pletion rates; 4) a review of the quality of the collected data for each method; 

and 5) a presentation of the costs of using each method. 
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The results presented in Section III should be relevant to most prime sponsors* 
^Figure B-l presents the distribution of the ratio of population to land area for all 
prime sponsors; that is, population density The figure shows that more than two- 
thirds of all prime sponsors encompass geographic areas that have fewer than 750 
people per square mile* Four of the five prim^sponsors included in this study have 
population densities in the same r^nge. At the opposite end of the scale, 19 percent 
of all prime sponsors have population densities, greater than 2,000 people per squat* 
mile. One prime sponsor included in this study has a population density in this 
range* 

I. Surrey Design 

For this study, approximately 800 participants were chosen for follow-up. Of 
these participants, roughly half terminated from CETA Title II funded programs 
* operated in Region VI and half from programs operated in Region VII. Based upon 
regional enrollment m figures for Title II, thfc sample was designed so that 75 percent 
of the participants terminated from Title IIBC and 25 percent from Title IID. In 
precise terms, 784 participants were included in the follow-up sample. Of the 784 
.participants, 395 were individuals who participated in programs operated in Region 
VI and 389 participated in programs operated in Region VII. 

The Region VI component of the research was undertaken by the Center for the 
Stndy of Human Resources of the University of Texas at Austin. As is shewn in" 
Table B-l, the participants included in the survey terminated during July and 
August of 1980 from Title II programs operated by the Capital Area Manpower Consortium 
and the Alamo Manpower Consortium. 

The original intent of the stiidy was to follow-up participants who terminated 
during August 1980 six months after their termination. However, the two prime 
sponsors involved in Region VI did not have sufficient numbers of August_t;erminees 
between them to result in a satsifactory distribution. The Capital AreA Manpower 
Consprtium had 80 Title* II terminees in August, and the Alamo Manpower Consortium 
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f; Figure B-l 

Distribution of Population per Square Mile 
by Prime Sponsor in 1980 



5.1 % 



4.3 



2.6 



1.9! 



1.7 



500 



1000 



1500 



2000 



Population per Square Mile 



Capital Area Manpower Consortium 
Alamo Manpower Consortium 
Kansas City Area Manpower Consortium 
Balance of Jackson County 
Independence City ^ 



85 
101 

260 
525 
2324 
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Table B-l: Distribution of Participants by Prime Sponsor, 
* Month, and Title Subpart of Termination 



Begion VI Begion VII 

Kansas City 

Capital Area Alamo Area Percent Percent 

Manpower Manpower Manpower * Jackson by By Title 

Consortium Consortium Consortium Independence County Total Month Subpart 

> 

Title IIB/C 

June v 
1980 - - - ~ 30 30 5.1 

July ' 

1980 33 - . 71 - 16 15 135 * 23.1 

August 

1980 , V 56 206 82 50 , 26 420 71.8 

Total for 
Title II 

B/C * 89 206 153 66 71 585 100.0 74.6 

Percent # & 

by Prime 

Sponsor 15.2 35.2 26.1 11.3 12.1 100.0 

Title 13 



July 

1980 19 47 4 2 72 36.2 

August 

1980 24 57 39 2 5 127 63.8 



/ 



Total for 

T* tie ^^ifi^ 

HO 43 57? 86 6 7 ^ 199 100.0 25.4 

Percent 
by 

Prime 

Sponsor 21.6 28.6 43.2 3.0 3.5 100.0 

r 

Total . 132 263 239 • 72 78 784 100.0 

Percent 
by 

Prime „ < 

Sponsor 16.8 33.5 30.5 9.2 9.9 100.0 

* • 
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had. 519. If a sample of 320 had been/lrawn from the Alamo Manpower Consortium and 
added to the 80 terminees from the Capital Area Manpower Consortium to make up the 
400 terminees needed for, the study, it was felt that the Alamo Manpower Consortium 
would have dominated any effects observed in the data. Instead, all 80 August 
terminees and all 52 July terminees were included in the sample for the Capital 
Area Manpower Consortium. Thus, a 100 percent sample of July and August terminees 
was taken for this prime sponsor. For the Alamo Manpower Consortium, 206 terminees 
were randomly selected from the^ 430 Title IIB terminees, and 57 terminees were 
randomly selected from the 89 Title IID terminees, resulting in a total sample of ■ 
395 individuals from Region VI. These" final figures are reflected in Table B-l. 

The .second component of the study included 389 individuals who terminated from 
programs operated in Region VII • This component of the study was undertaken by 
Human Resources Data Systems, Inc. As is ,shown in Table B-l, -the participants in- 
volved terminated during June, July and August of 1980 from Title II programs 
operated by the Kansas City Area Manpower Consortium, Independence,, and Jackson 
CounVy Prime Sponsors. 



As Ui 



was the case .with Region VI, in Region VII there were not enough terminees 
in August 1980 to create a sample of sufficient size for the study. Therefore, 
all terminees from the month of July 1980 were included in the sample along with 
all the August terminees. In addition, 30 participants who terminated during June 
1980 from programs operated by the Jackson County Prime Sponsor were included % as 
• well. 

During January 1981, participant information: was collected from the Manage- 
ment Information Systems (MIS) of the five prime sponsors. The Participant Informa- 
tion Form (PIF) contained in Appendix F was used for this purpose. All of the data 
elements listed' on the PIF were sought for each participant. Once each of 784 
participants included in the study was identified, and the PIF for the indiv^ual^ 
completeji, the forms were placed in numerical order according to the identification 
numbers assigned to the participants by the prime, sponsor. Those who were to be 
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interviewed by the telephone method were selected by\choosinfe every third jg^rticl- 
pant beginning with th^fcjfirst participant. Those who Were to be interviewed by the 
personal visit method were selected by choosing every third participant beginning 
with the second participant. And finally, those participants who were to be part 
of the mail survey w£re selected by choosing every third participant beginning with 
the third participant. . 

Table B-2 presents four demographic characteristics for all of the participants 
selected for illusion in the study, by contact method. From the table, it can be' 

seen that, generally, the characteristics of participants in each of the subsamples« 

/ 

are the same. While there is some variation across methods, the chi-square test of 
significance has been applied for each' of the four characteristics, and, in every 
case, it has been concluded that there is no reason to reject the null hypothesis 
of independence at reasonable levels of significance. One place where Isome varia- 
tion which is mildly disturbing does occur, is the education characteristic. m For 

/ x 
the/ total sample, 16 percent of the participants have completed 13 or more £ears 

of /education. For the telephone subsample, 22 percent "have achieved this level of 

lucation, whereas for the personal visit subsample, only 12.5 percent of the par- 

'icipants have the same level of achievement. 



II. Location, Contact and Interview Strategies 

At the end of January 1981, each participant included in the sample was sent 

J K 
a letter explaining the purpose of the upcoming interview and stressing its 

voluntary .and confidential nature. Interviews were undertaken in February 1981 

and used the telephone interview method, the personal visit interview method and 

the mail survey method. Bach of these methods is outlined in this s&ction. 



r 
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Table B-2: Demographic Characteristics of 
Participants by Contact Method 





Telephone 


Personal Visit 


Mail Survey 


Total 




Number 


Percent 


Number 


Percent 


Number 


Percent 


Nunujer 


percent. 


Age 








• 










16-19 


36 


14.1 


36 , 


14.3 


39 


14.6 


.111 


14.3 


20-24 


106 


41.6 


101 


40.2^ 


117 • 


43.7 


324 


41.9 


25-44 




37. 0 


yy 


jy • ** 


i m 

1U1 


17 7 




38 1 


45-64 


18 


7.1 


15 


6.0 


11 


4.1 


44 


5.7 , 


Total 




100 . 0 


251 


100 . 0 


268 


1UU . u 


11 A 




Years of Education 




» 






• 








Less than 8 


21 


8.1 


23 


9.0 


25 


9.3 


69 


8 . o 


9-11 


• 90 


34.9 


99 


38.7 


106 


39.6 


295 


37.7 


12 " 


90 


34.9 


102 


' 39.8 


99 '* 


36.9 . 


291 


37.2 


13 or more 


57 


22.1 


. 32 


12.5 


38 


14.2 


127 


16.2 


Total 


258 


100.0 


256 


100.0 


• 268 


100. 0 


782 


100. 0 


* 

Sex 


















Male 


118 


.45.7 


114 


44.5 * 


128 


47.4 


360 


45 ;9 


Female 


140 


54.3 


142 


.55.5 


142 


52.6 


424 


54.1 


Total 


258 


jvLOO.O 


256 


100.0 


270 


100.0 

OS 


784 


100.0 . 



Race 



Black 


9X 


35.3 


91 


35.5 


107 


39.6 


289 


36.9 


White 


93 


36.0 


91 


35.5 


85 


35.5 


269 


34.3 


Hispanic 


69 


26.7 


73 


28.5 


72 


26.7 


214 


27.3 


Other* 


5 


2.0 


1 


0,4 . 


6 


2.2 


12 


i:5 


Total 
* 


258 


100.0 


256 


100.0 


270 


loo.o y 


^784 


, 100.0 


♦Includes American 


Indian, 


Alaskan, 


Asian and Pacific 


Islander. 
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Telephone Interview Method ' ^ 

At the beginning of February, interviewers initiated the process of contacting 
and interviewing epch of those participants who had been assigned to the telephone 
interview subsample. At the outset, interviewers called participants between 
4:00 p.m. and 8:00 p.m. at the telephone numbers which were available from the 
prime sponsor MIS* The initial call led to one of two possible outcomes. First, 
in some cases, the participant was immediately contacted and was then asked to 
complete the interview ^If it w^fe not possible at that time for the participant 
to complete the interview, arrangements were made to call back at a time which 
was more convenient for the participant. Second, in many other cases, the partici- 
pant was not immediately contacted and a variety of location and contact methods 

6 

was then employed in an attempt to locate She participant. 

The methods which were used to locate and contact members of the telephone 
interview subsample can be grouped into five categories. First, callbacks were made 
for a few consecutive evenings. If contact vas not made, the participant was called 
back at various times during the day f or % a few consecutive days.. Second, for those 
participants who had given the prime sponsor the name and telephone number of a • 
friend or relative, the contact person was called in an attempt to ascertain the 
location of the participant. Third, when more current 'information was required, the 
local telephone directory was utilized, and, when necessary, directory assistance 
was also contacted. Fourth, when * the prime sponsor MIS indicated that the pdttici- 
pant had been job placed at termination, if the information was available; the ^ 
employer with whom the participant was placed was contacted. In some cases, the 
participant was still employed by this employer and arrangements could be made to 
complete an interview. If 'the participant had left this employer, a forwarding 
address or telephone number was sought. Finally, a cross- listing city directory 
was used, when necessary, to identify neighbors who might know-how to contact the 
participant. 

>s B-8 - 
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One location and contact method which interviewers did not haye- the option 
of utilizing with the telephone, interview subsample, was to actually travel to the 
most current address available for the participant* Interviewers were limited to 
location and contact methods which involved the use of the telephone as the primary 
resource and minimized the use of travel* Finally, it should be clear that if, at 
any point during the search process, a participant was located and contacted, an 
offer to c^plete an interview was made* If it was not then possible to complete 
the .interview, arrangements were made to call back at a more convenient time. 
Telephone intervieyers used the questionnaire contained in Appendix C. 

Personal Visit Interview Method J 

For participants who had been assigned to the personal visit interview sub- 
sample, the procedures utilized to locate and contact the participants were vir- 
tually identical to those outlined for the telephone interview method. However, 
the personal visit "interview method also ha<f available to it an additional location 
and contact method. It was permitted for. project staff to pursue a contact^ through 
field work. In other words, if a preliminary phone contact was not made, the in- 
terviewer could visit the^ddress given by the participant. In cases in which par- 
ticipants were not located at the giv<*n address, ^the interviewers could use ,the 
opportunity to ask neighbors for potential leads as to the participants 1 whereabouts 

Aside from the- additional search procedures, the personal visit interview 
method also provides that interviews be conducted in person. Thus, once the parti- 
cipant has been located and contacted by telephone, an appointment is made fcfr a 
time when the interview can be. administered. The in-person interview utilizes the 
same questionnaire as the telephone interview. 

4 _ 
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Mail Survey Methpd 

^ < — r 

For those participants who were assigned to the' mail survey subsample, a 
cover lettet and the survey instrument were sent to the most recent address recorded 
in the MIS, on or about January 26, 1981. As completed mail survey questionnaires 
were returned, each was reviewed to determine which responses were usable and which 
were riotl *It was then, necessary to code the responses in a manner consistent with 
the coding of the telephone and personal yisit interviews. For questionnaires 
which were returned and not completed because the questionnaire could not be 
delivered by the postal service, other location methods were employed. These methods 
included city telephone directories, cross-listing city directories, neighbors and 
employers. Interviewers, however, did not have the option of using two of the 
location and contact methods. That is, interviewers could not attempt to reach the 
participant by telephone or actually travel to the most current address available 
for the participant. 

* * 

A second mailing was undertaken on or about February 13. Those participants 
who did not return completed questionnaires by that date, or whose questionnaires 
were returned as undeliverable but for whom new addresses had be£n obtained, were v 
sent a second set of mail, survey materials. ^ 

As each interview was completed, in both the telephone and personal visit 
methods, it was checked and coded by the interviewer. Each completed mail ques- 
tionnaire was al^o reviewed and edited by the survey managers at each of the two - 
sites. Finally, the data were entered into a computer system, cleaned, and then 
"transmitted to the Massachusetts Department of Manpower Development. 



'III. Findings of the Field Work 

This section-presents findings regarding the use of each of the three-purvey 
methods. The discussion of the results is divided into five parts. First, the 
demographic characteristics of those participants who completed an interview are 
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presented. Second, the proportion of participants who were contacted by each 

method and the proportion of participants who, completed interviews by each mettrad 

are calculated, ^jghe location <and t interview strategies used i$ each method are 

analyzed to determine the reasons for some participants not having completed the 

interviews.** Third, the determinants of the contacj^and completion rates are 

studied in detail. Initially; both contact and completion rates are br'oken down 

by demographic and program characteristics. This analysis is extended by employing 

multiple regression analysis to identify the separate effects individual demographic 

,and program characteristics have on contact and completion rates. Fourth, the com- 
, # 
♦ • • 

pleteness and consistency of the data are examined for each of the three methods by 
identifying ^certain checkpoints within the .questionnaires which allow for analysis 
to determine which of t)ie- methods led to the applicable sections and questions 

being completed most, of ten. . # Fif th and finally, the costs of locating, contacting 

'"fr- 
aud interviewing participants by each method, as Wtell as cpding and checking com- 
pleted questionnaires, s are calculated. This cost' analysis, when combined with much 
of the previous* analysis, permits * some determination of the cost-effectiveness of 
the alternative survey methods to be made, 

* 

Demographic Characteristics of Respond ents 

Table B-3 presents the demographic characteristics of those participants who 
completed an interview or a questionnaire. The table shows that the age,* sex and 
jr^ce characteristics of those who responded are very similaif for each of the three ( 

contact methods. It is only the education characteristic where significant deviaj- 

1, 

tion occurs. Twenty-two percent of the ^respondents in the telephone interview 
subsample had ' 13 or more years ' of education. By comparison, only ten percent of 
the respondents in the personal visit and mail surveys had 13 or more years of 
education. Thus , ; if a prime sponsor elects, to use. the telephone interview method , 
'i£ may be that, a more highly educated sample will respond relative to the personal 
* visit and mail Survey methods. Since more highly 'educated individuals tend to 
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Table B-3 : Demographic Characteristics of Respondents by Contact Method 



Telephone 
Number Percent 



Personal Visit 
Number Percent 



' Mail Survey Total 
Number Percent Number Percent 



Age 



16-19 


19 


16.5 


16 


15.7 


9 


18.0 


44 


20-24 


40 


34.8 


38 ' 


37.3 


21 


42.0 


. 99 


25-44 


49 


42.6 


40 


39. 2 


15 


30.0 


104 


45+ 


7 


6.1 


, 8 


7.8 


5 


10.0 


20* 


Total 


115 


100.0' 


102 


' 100. 0 


50 , 


loo.o 


267 


c 

Years of Education 












\ 




8 or Less 


5 


4.3 


12 


11.8* 


3 


5.9 


20 


9-11 


41 


35.3 


32 


31.4 


22 


43.1 


95 


12 


44 


37.9 


47 


46.1 


' 21 


41.2 


112 


13 or More 


26 


22.4 ' 


11 


10.8 


5 


9.8 


42 


Total 


116 


100.0 


102 


100.0 


'51 


100.0 

r 


269 


Sex 
















Male 


44 


37.9 


48 


47.1 


19 


* 

37.3 


111 


Female 


72 


62.1 * 


54 


52.9 


32 


62.7 


. 158 


Total 1 


116 


100.0 


, 102 


100.0 


51 


100.0 


269 



16.5 
37.1ry 
39.0 1 
7.5 
100.0 



7.4 
35.3 
41.6. 

.slS. 6" 

l&b.o 



41.3 
58.7 
100.0 



Race 

Black 

White 

Hispanic 

Other* 

Total 



35 


. 30.2. 


35 


34.3 


16 


42 


36.2 


37 


36.3 


23 


39 


33 t 6 


29 


28.4 


12 






1 


1.0 




116 


100.0 


102 


100.0 


51 



31.4 
45.1 

100.0 



<86 
102* 
20, 
1 

269 



32.0 
37.9 
29.7 
0.4 
100.0 



♦Includes American Indian/ Alaskan, 
Significant at the 10% level 



Asian and Pacific Islander 
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experience greater labor market success, the telephone method could lead to 'a 
more opt im istic view of program outcomes than would be obtained if one of the 
alternative methods was chosen* 

* 

Contact and Completion Races 

A primary issue of concern is the nature and magnitude of the response of par- 
ticipants' to each survey method. Table B-4 shows that the proportion of participants 
who were contacted and the proportion who completed interviews vary by the contact 
method. The table shows that 48 percent of the participants were contacted in both 
the telephone and personal visit, surveys . However, in the telephone method case, 
45 percent of the participants completed the interview, while in the personal visit 
Interview case, only .40 percent of the participants completed the interview. Members 
of Che mail survey subsample are assumed to have been contapteth if the-Postal 
Service did not return the mailed questionnaire as undeliverabie. Therefore, as 



would be expected, a relatively large proportion of the mail survey subsample was 
eonsidered to be contacted, /an 80 percent contact rate. However, as is frequently 
the- case with mail survey^,' the number of individuals who completed and returned 
the survey was small. Only 19 percent of the mail survey subsample completed and 
returned a questionnaire. * 

One might expefct the' contact rate for those participants in the personal 
visit subsample to have been higher than for those participants in the telephone 
interview subsample.* This .difference would arise because interviewers had the 
additional alternative of visiting the address of the participant to attempt an 
interview." In'practice, however, interviewers were very reluctant to visit an 
address without previously having made an appointii&QC^^Ja fact, this alternative 
was employed only infrequently. Therefore, the contact rate was virtually 

4 1 

identical for both the telephone and personal visit methods. 
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Table B-4: Contact and Completion Rates , How Contacted and Reason for 
Non -Completion by Contact Method 



^Telephone 
Number Percent 



Contact Rate a 


123 


47.7 


124 


Completion Rate a 


116 


45:0 


aoi' 


How Contacted 








Intake Phone No. 


' 109 


85.8 


101 


Directory Assist* 


2 


,1.5 


2 


Went to Home 






3 


Employer 


3 


2.4 


1 


Prime Sponsor 


6 


4.7 


9. 


" Relatives, -Friends, 








Neighbors 


6 


4.7 


7 


Mail 








Other 


1 


0.8 


2 


Total 


127 


100.0 


125 



Reason for Non - 
Cdmfiretion — 

/ 

No Phone No. on Intake 
Wrong No. on Intake 
Disconnected Pnone No. 
Unpublished Phone No. 
Out of Stat*^Phone - 
Moved, No Address 
Could Not Find^ 
Appointment Not Kept 
• Refused to Answer 
Refused to Complete 
Could Not Complete 
Language Problem 
Mail* 
Other 
Total 



Personal Visit 
Number Percent 



48.4 
39.5 



Mail Survey : Total 
Number Percent Number Percent 



224 
51 



83.9 
18.9 




100.0 



100.0 



100.0 



10 


3.9 


5 


2.0 






24 


9.3 


19" 


7.4 






22 


8.5 


27 


10.5 






4 


1.6 


3 


1.2 






1 


0.4 


0 


O.O 






25 


9.7 


28 


10.9 






39 


15.1 


34 


13.3 






o • 


0.0 


8 


3.1 






5 


1.9 


10 


3.9 








t 0.0 


$ 


2.0 






; i 


£ 0.4 


0 


0.0 






1 


0.4 


0 


0.0 














219 


100.0 


10 


3.9 ■ 


16 


6.3 






142 


100.0 


155 


100.0 


219 


100.0 



471 

V 

268 



210 
4 
3 
4 

.15 

13 
224 
3' 



60.1 
34.2 



4411 
0.8 
0.6 
0.8 
3.1 

2.7 
47.0 
0.6 
100 .-o 



15 


2.19 


43 


8.3 


49 


9.5 


7 


1.3 


1 


0.2 


53 


10.2 


73 


■ 14.1 


8 


1.5 


15 


2.9 


5 


1.0 


1 


0.2 


1 


0.2 


219 


42.4' 


26 


5.6 


516 


100.0 



•The reason mail survey subsample members 'did not complete a questionnaire is unknown. 
a Chi-Square test significant at the 1% level. 
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Table B-4 also shows the sources of information which were used to^ obtain 
a contact with the participants* Of those individuals contacted by either the' tele- 
phone or the personal visit method, over 80 percent of the contacts were made by 
means of c the telephone number obtained from the prime sponsor files. Such a result 
is important evidence supporting the notion that one very productive step which 
p^ime sponsors can take to potentially infprove contact rates .is to obtain one or 
more current telephone numbers during an exit interview at termination. In fact, 
95 percent of the participants contacted by either method were contacted by means 
of the telephone nu)taer obtained from the prime sponsor file, by means of contact 
with friends or relatives or by means of other sources of information available 
from the prime sponsor. Thus, obtaining a current telephone number during an exit 
interview both for the participant and a friend or relative, as well as making a 
concerted effort to gather all information about each participant available through- 
out the prime sponsor, should enhance the ability of prime sponsors to locate and 
contact participants* 

Finally, Table B-4 presents the reasons for not having completed interviews 
with certain participants, j/rom the top of the table, it can be seen that the com- 
pletion rate is much lower/for the personal visit subs-ample than for the telephone 
interview subsample. The reasons for non-completion, also shown in the table, in- 
dicate that the difference in the completion rates results fronf members of the 
personal visit interview subsample refusing to complete an interview by failing to 
keep an appointment, refusing to answer, or refusing to complete the interview. 
Apparently, participants prefer the convenience and relative anonymity that the 
telephone interview allows. 

Determinants of the Contact and Completion Rates 

In addition to studying the contact and completion rates by survey method, it 
is also of interest to examine the manner in which contact and completion rates vary 
by demographic and program characteristics. Table B-5 shows that of the seven 



Table B-5: Cocpletion and Contact Rate by Demographic and Program Characteristics 



Contacted* 



Number 



Percent 



Conpleted Interview 
Number Percent 



Total 



247 



48,1 



268 



34.2 



Age 

16-19 
20-24 
25-44 
45+ 

. Education 

8 or Less 

9-11 

12 

13 or More 



98 
109 
18 



18 
85 
104 
40 



52.8 
44.1 
51.2 
51.4 



40.9 
45.0 
?4.2 
44^9 



18 
88 
99 
17 



20 
96 
112 
40 



40.0 
33.1 
36.0 
47.2 



28.2 
32.5 
38.5 
31.5 



Sex 

Male 
Female 

Race 

Black 
White 
Hispanic 

Limited English 
Speaking Ability 

Yes 
No 



99 
148 



77 
90 
78 



5 

242 



42. 7^ 
52.5 



42.3 
48.9 
54.9 



38.5 
48.3 



109 
159 



85 
104 
78 



5 

262 



30.3 
37.5 



31. 7 C 

38.7 

36.4 



20.0 
34.6 



Number of Dependents^ 



Zero 

One 

Two 

Three 

Four 

Five 

Six 



62 
74 
42 
28 
11 
16 
14 



49.2 
46.0 
48.8 
44.4 
44.0 
69.6 
46.7 



67 

77 

52^ 

30 

11 

14 

17 



35.3 
30.7 
36.6. 
33.0 
26.8 
56.0 
38.6 



Aid to Families with Dependent 
Children 

Yes 
No 



47 

200 



55.3 
46.6 



49 
219 



39.5 
33.2 
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Table B-5: (continued) 



Contacted* 



Number 



Percent 



Completed Interview 
Number Percent 



Region 

VI 
VII 



132 
115 



50-2 
45-8 



132 
136 



33.4 
35.0 



Activity 

On-The-Job Training 
^ Occupational Training 
Public Service Employment 
Other 



26 
93 
70 
58 



48.1 
46.7 
53.0 
45.3 



21 
103 
74 
68 



26.9 
34.0 
37.2 
33.8 



Title II Subpart 

IIBC 
IID 



177 
70 



46.3 
53.0 



195 
73 



33, 
36, 



Placement 



Yes 

NO- 



84 
163 



65.1 
42.3 



76 
192 



44.7 
31.3 



*Only the telephone and personal visit interview subsamples are included in the calculation 
of the contact rate. 
a Chi-Square test significant at the 1% level 
b Chi-Square test significant at the 5% level 
c Chi -Square test significant at the 10% level 
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demographic characteristics -used, tile contact rate is systematically related otOy^ 
to the sex of the participant. That is, the table shows that women tend to be 
easier to contact than men.* The table ^also shows that of the four program charac- 
teristics listed* the contact rate is systematically related only to placement. 
With regard to the interview completion rate, two demographic characteristics are 
of importance. Women tend to complete interviews more often and blacks tend to 
complete interviews less often. In addition, as is the case with the contact rate, 
the completion rate is higher for those participants who were placed at termination. 
Since it is very likely that the placement rate and some demographic characteristics 
are closely related, the use of multivariate statistical techniques is appropriate. 
To this end, two sets of linear regression equations have been estimated. 
first set of equations attempts to identify the determinants of the contact rate. 



The second set of equations attempts to identify the determinants of the completion 
rate. 

Table B-6 presents the hypothesized relationships between the independent vari- 
ables and the dependent variables, the contact rate and the completion rate. For 
the first set of equations, the dependent variable has a value of one if the parti- 
cipant was contacted and zero if not. The data which have been used td estimate 
this set of equations are combined for the telephone and personal visit subsamples. 
Due to the inability to verify contact in a mail survey, the raaU survey subsample 
has been excluded for this portion of the analysis. It is hypothesized that the 
personal visit interview method will lead to signif idantly more participant contacts. 
For the second set of equations, „ the dependent variable has a value of one if the 
participant completed an interview and zero otherwise. The data which have been^used 
to estimate this second set of equations include all three subsamples. In this case, 
it is hypothesized that the personal visit interview method will lead to more completed 
interviews and the mail survey method will lead to fewer completed interviews . 
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Table Hypothesized Relationships Between Contact and Completion 

Rates and Demographic and Program Characteristics 



Method 

Telephone 
Mail 

Personal Visit 

A2e 

16-19 
" 20-24 
25-44 
45+ 

Years of Education Completed 
Sex 



Contact 



Base 

Not Applicable 
+ 



* 

. Base 



Interviewjdompletion 



Base 
■ + 



Mala 
Female 



Base 



Race 

White 4 " 

Black 

Hispanic 

Limited English 

Yes 

No 1 

No. of Dependents 

AFDC 

Yes 
No 

Placement 
Yes 

No k 



Base 



Base 
+ 



Base 



+ 

Base 



Region 

VI 
VII 



* 

Base 



ERIC 



+ Variable positively related to dependent variable. 

- Variable negatively related to dependent varaable. 

* Relationship between variable and dependent /variable indeterminate, 
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For the remaining independent variables, it is hypothesized that participants 

in the 16-to-19 and 45-and-over age groups will be contacted less frequently and 

complete interviews less often than participants *ia the 25-to-44 age group. No 

hypothesis is advanced regarding the behavior of those in the 20-to*-24 age grpup 

relative to members of the 25^:0-44 age group. For those participants with higher 

levels of education it is thought to be easier to achieve contact and to .obtain a 

completed interview because of a greater familiarity with research generally and 

survey research in particular. No hypothesis is advanced about the relationship 

between the sex or race of the participant and contact or completion. Because of 

the presence of language problems, these with limited english speaking ability are 

thought to be more difficult to contact and interview. Given greater child and 

home care responsibilities, those participants with more dependents' are hypothesised 

to be easier .to contact and interview. The incentive structure of the AFDC system 

creates a situation in which recipients are assumed to be at home with greater 

1 v ^^^^^^ 

frequency and thereby may be easier to c^fnYact and interview. Those participants 

who are placed in unsubsidized employment by the prime sponsor generally remain in 
a relatively compact geographic area and, therefore, are more frequently contacted 
and interviewed. Finally, no hypothesis is advanced regarding differences which 
may exist across regions or the sites which conducted the field work. 

Table B-7 presents the results of estimating a set of equations which attempts 
d^to identify the determinants of participant contact. In the first equation, the 
only independent^'ariable included Is a variable which has a value of one if the 
participant was a member of the personal visit interview subsample and zero other- 
wise. The results presented in Table B-7* show that there is no significant 

^ s . 

difference between the contact rates of the telephone and the personal visit inter- 

. view subsamples. 
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Personal Visit 
Age 16-19 
Age 20*24 
Age 45+ 

Years of Education 

Sex 

Black 

Hispanic 

Limited English 

Mo. of Dependents t 

AFDC' 

Placement 

Region 

Constant 



R 2 



Degrees of Freedom 



Recression Analysis for Participant Contact 


* 

— * 




i 

(1) 


(2) 


(3) 


> 

(4) 


* 

(5) 


0.01 


0.01 


0.01 


0.01 


0.01 


(0, 04) 


(0,04) 


(0. 04) 


(0.04) 


(0.04) 


- 


0,04 


0.05 


0.ff4 


0.03 




(0, 09) 


(0. 09) 


(0.09) 


(0.09) 


- 


-0,05 


-0.04 


-0.05 


-0.06 




(0.05) 


(0.05) 


(0.05) 


(0.05) 


- 


0,02 


0.03 


0.02 


0.03 




(0.^)9) 


(0. 09) 


(0.09) 


(0.09^ 


- 


0,01 


0.01 


0.01 


0.01 




(0.0L) 


(0. 01) 


(0.01) 


(0.01) 


- 


-0.08 b 


-0.08 b 


-0.08 b 


-0.08 b 




(0.04) 


(0.Q4) 


(0.05) 


(0.04) 




-0.07 C 


-0.07 C 


-0.07° 


-0.06 


■ 


(0.05) 


(0.05) 


(0.05) 


(0.05) 




'o.08 C 


-0.01 


0.07 


0.04 




(0.06) 


(0. 06) 

* 


(0.07) 


(0.07) 




-0.19 C 


, -0.17 , 


-0.19 


-0.19 C 




(0.15) & 


(0.14) 


(0.15) 


00.14) 


- 


< 

0.02 C JjSf 


0.02 b ' 


0.02 C - 


0.02 C 




(o. oi)Wtf\ 


(0.01) 


(0.01) 


(0.01) 




0. 04 


0.05 


, 0.04 


0.06 




(0.06) 


, (9. 06) 


(0.O6) 


(0.06) 


- 


- 


\ 6.24* 




0.29° 




* 


J (0.05) 




(0.06) 






- - 


0.01 


-0.12 b 






1 


(0.05) 


(0.06) 


0.48 a 


0.33 b 


0.31 b 


b 


' b 
0.36° 


(0-03) 


(0.16) 


(0.16) 


(0.16) 


(0.16) 


0.0 




.04 


.01 


.05 


- 0,46 


1.52 


2.98 a 


1.39 


3.06 


1/501 


11/491 


12/490 


12/490 


13/489 



Significant at the 1% level. 

Significant at the 5% level. * 
^Significant at the 10% level. 

Standard errors of the coefficient estimated are in parenthesis. Dependent variable has a 
value of one if the participant has been contacted and zero otherwise. Coefficient estimates 
have been rounded t>o th* nearest one hundredth for ease o£ presentation. 
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The second equation presented in Table B-7 introduces a series of demographic 
variables into the equation. The first result to note is that the estimate of the 
coefficient on the personal visit variable did not change with the introduction of 
the additional variables. This confirms the result that the contact method is not 
related to the participant's demographic characteristics. The second result to 
note is that the estimates of the c&ef ficients on the age and education variables 
are not significantly different from zero. The results also show that females, 

•r * 

Hispanics and participants with a large number of dependents are more likely to be 
contacted, while blacks and participants with limited english, speaking ability are 
less likely to be contacted. ^ 

One reason for having undertaken the estimation of these equations is the 
realization that being placed in unsubsidi 2 ed employment is not only related to a 
number of demographic variable3"&ut~-atso 'ttr having been contacted for an interview. 
Thus, to examine this issue, a third equation has been estimated .which includes a 
variable which has a value of one for participants who are placed^and zero other- 
wise. Column 3 of Table B-7 indicates that the inclusion of the placement variable 
leads to three interesting results. First, the personal visit method is not more 
likely to contact participants. Second, the coefficient estimates dn the Hispanic 
and limited english variables are now no longer significantly different from zero. 
This- is because Hispanics tend to be placed more often and those with limited 
english speaking ability less often. Thus, in the absence of the placement term, 
these variables were capturing the effect of placement on contact. Third, it can 
be seen that the coefficient on the placement variable is positive and significantly 

different from zero. » • f 

An additional reason for having undertaken tKe estimation of these equations is 
' the-possibility that the region in which the project operated has an Important 
effect. Column 4 of Table B-7 shows that when the placement variable is taken out 
and the region variable, which has a value of one if the participant was in Region 
VI and zero if the participant was in Region VII, is added, the coefficient 'estimates 
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3 ■ 

If 

are nearly identical to those presented in Column 3. This is because the region 
variable 7 is relatively highly correlated with the placement variable* Thus, in 
the absence of the placement variable, the region variable attempts to play the 
Same role* 

Finally, Column 5 of Table B-7 presents an estimate in which" both the placement 
and the region variable are entered* Again, it can be seen that there is no 
difference between the telephone and personal visit , interview taethod. In addition, 
females , -participants with limited english or a larger number > of dependents, those 
who are placed and those in Region VII are more likely to have been contacted. 
Because blacks tend to be placed in unsubsidized employment less often and are more 
likely to be found in the Region VII part of the sample, when the placement and 
region ^variables are entered into the equation, the black variable has no separate 
influence. * 

In addition to the above analysis,* the same set of equations has also been 
estimated when the dependent variable is the completion status o'f the, interview for 
the participant. First, Table B-8 shows that, -relative to members of the telephone 
subsample, the members of the mail survey subsample are less likely to complete an 
interview, and members of the personal visit subsample are about as likely to com- 
plete an interview* Second, the sex and the race of the participant play a much 
Less important role in completing an interview than they did in initially contacting 
the participant. However, as was the case with the previous set of equations, partici- 
pants with limited english speaking ability are less likely to complete an inter- 
view and those'with more dependents are more likely to 'complete an interview. Such 
a set of results is not unreasonable. ^^While they may be more or less difficult to 




• contact or locate, males* females, whiles, 'blacks and Hispanic^, once contacted, 
are, all equally willing to» be interviewed. Limited english speaking ability is an 
impediment not only to c6ntact but also to interview completion. Although it is 
not intuitively obvious why, it appears that individuals with larger numbers of 
dependents are not only easier to contact but are also more likely to complete 
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Table B-8: Regression Analysis for Interview Completion 





(1) 


(2). 


(3) 


(4) 


(5) 


Mail 


-0.42 a 


-0.42 a 


y , -0.42 a 


-0.41 a 


-0.38 a 




(0. 05) 


(0*05) 


(0.05) 


(0.05) 


(0.06) 


Personal Vi^it 


-0. 05 


-0.05 


-0.05 


-0.05 


-0.05 


(0. 04) 


(0.04) 


(0.04) 


(O.04) 


(0.04) / 


Age 16-19 




* 

0.02 


0.03 


0.02 


0.01 




(0.07) 


(0,08) 


(0.07) 


(0.07) 


Acre 20-24 




-0.04 


-0.03 


-0.04 


-0.04 




(0.^4) 


(0.04) ' 


(0.04) 


(0.04) 


Age 45+ 




0.05 


0.06 


0.D5 


0.06 > 




(0.08) 


(0.08) 


(0.08) 


(0.08) 


Years of Education 


i 


0.01 


0.01 


0.01 , 


C.01 






(0.01) 


(0.01) 


(0.01} 


(0.01) 


Sex 




-0.05 C 


-0.05 


-0.05 C 


-0.05 






(0.04) 


(0.04) 


(0.04) 


(0.04) 


Black 




-0.06 C 


-0.07 c 


-0.06 


-0.05 




• 


(0.05) 


(0.04) 

* 


(0.05) 


(0.05) 






0.04 


-0.01 


0.05 


0.04 


<* 




(0.05) 


(0.-05) 


(0.05) 


(0.05) 


T*i mi t*_Ad Enolish 




-0.17° 


-0.17° 


-0.17° 


-0.19° 


♦ 


(0.12) 


(0.12) 


(0.12) 


(0.12) 


No* of Dependent! 


m 


0.01 C 


0.02 b 


c 

0.01 


O.&f 




(0.01) 


(0.01) 


(0.01) 


(0.01) 


afhc* 




0. 03 


0.03 


0.03 


0.04 






(0.05) 


(0.05) 


,(0. 05) 


M0. 05) 


Ola wAwnw ♦» 






0.14 a 




a 

0'.19 








(0.04) 




(0.05) 












b 


Region 








-0.02 


-0.11 








<0.05) 


(0.05) 


Cons tant 


0.45 a 


0.36 a 


0.35 a ' 


0.37 a 


0.39 a 


^ 


(0.03) 


(0.13) 


(0. 13) 


(0.14) 


> (0.13) 


R 2 


.10 


.10 


.12 


.10 ■ 


.12 


F 


34.57 a 


6.82 a 


7.20 


6.31 


7.05 


Degrees of Freedom 


2/611 ' 


12/601 


13/600 


13/600 


14/599 



"^significant at the 1% level. " ✓ 
^Significant at the 5% level. 
Significant at the 10% level. 

Standard errors of the coefficient estimates are in parenthesis. Dependent variable has a 
value of one if the participant completed the interview and zero otherwise. Coefficient ^ 

© timates have been rounded to the nearest one hundredth for ease of presentation. 
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*he interview. Third, as* is the case .with the first set of equations, those par- 
ticipants who were placed in' unsubsidized employment and Chose in Region VII are 
mote likely to have completed an interview. ~ t 

Data Quality ' v ' } „ ^ ^ 

- , Another issue of 'primary concern is the quality, of the data' which.are generated 
by each of the three survey methods. Data quality in the present context refers to 
the consistency and completeness of the -data set which results from the use of any 
one of > the three contact methods.^ It does not refer to quality in the sense that 
the data accurately reflect the true values of the variables for each individual , 

as such true values are not known. Rather, the purpose of this section is to make 

4 < t . ' 

an assessment, of the use bf^the quest idnnaires with each -of the three methods. 

Table B-9; presents schematically the eight checks of completeness J and con-- - 
sistency to which the data have been subjected. 'To illustrate the manner in which 
these checks were done, Table B-9 Indicates that choice No. 1 involves completing 
Sect^on-'III on the telephone and personal visit questionnaire or .questions, 2 to 5 
on. the mail survey questionnaire if the respondent was not working, at the time the 
program ended ,\ As. a check of completeness, the percent of those interviews in 
which. Section III or questions 2-5 should' have .been completed, but the interviewer 
chose -not to complete them, has been calculated. In short, the percent of -inter- 

• views 'which 'were' completed with an Incorrect choice'is calculated for each contact 
method:" " Similarly^ Table B-9 also' indicates that Skip No. 1 involves a skip to 

, the appropriate next question depending on the response ter question' 14 in the tele- 
phone and personal visit questionnaire or question 3 on the mail survey question- 
nalrte. Again, as a check of consistency, the percent -of those - interviews in which 
an incorrect skiR was "made has been calculated for each contact method . 
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Tafilo B-9: Plan for Checks of Consistency and Completeness 
of *Follow-Up Questionnaires 



Choice or Skip 

Number 

1 r ■ 

Choice No* 1 



ft 



Labor Market 
Situations 

Respondents who were not 
working at the time program 
ended 



Section or Question Number 



Telephone and Personal 
Visit Questionnaire 

Section III 



Mail Survey 
Questionnaire 

Q- 2-5, 



Choice No* 2 



Respondents who had one or 
more Jobs 



Section IV 



Q. 7- 16V 
17-23^ or 
24-32* 



Choice No* 3 



Choice No. 4 



Respondents who ha4 more 
than one job 

Respondents who were not 
world, ng at the time of 
the interview 



Section V 



Section VI 



/ 



Q. 17-23 or 
24-33 

Q. 34-39 



, Skip No. 1 



Skip No. 2 



Skip. No. 3 



Skip No. 4 



Respondents vho were not working ~ 14 

at the time program ended, * 

look for work every week, yes 

or no / 

Respondents who had one or 2l 

more jobs and who left their 

first job * 

Respondents who had more than * 2 5 

one job and who were not 
working at time of interview 

Respondents, who we'rd not • 28 
working at the 'time of the 
interview and were looking 
for work, yes or no 



15 or 31 



31 



36 



rr 
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Table B-10 presents the results of these eight checks. It can be seen that 
the personal visit method results in a more complete and consistent data set. 
However, relative to the data set produced by the telephone method, tjie difference 
in the quality of the personal visit data set is small . In seven of eight in- 
stances, the absolute number of errors c^iffers by only one, or not at all* 

The mail survey questionnaire, in contrast, resulted in a data set of sub- 
"st^ntially lower quality than -the other two. In order to test the ability of the 
mail survey to yield^a data set' which was identical to that generated by the tele- 
phone and personal visit questionnaire, a somewhat lengthy mail survey questionnaire 
was required. A close reading of the mail survey questionnaire will reveal that in 
order to simplify the instructions required to use the questionnaire, some questions 
are repeated two or three times, basically to allow fQr differences in the tenses 
of verbs. -In spite of these elaborate attempts at simplification, the error rate is* 
high for several of the checks 4one on the mail survey data. In particular, Table 
B-10 shows that in cases where more than' one* set of questions may apply, such as 
choice-No. 3 and skip No. 2, the error rate is notably higher. However, it is of 
some interest to note that the rate for choice No. 2 is not substantially 

higher than for either of the other two methods.' Without physically examining each 
questionnaire, it is not known whether those respondents who have held more than 
one job had greater difficulty completing the questionnaire or if the coding of 
responses undertaken by project staff members is the source of the difficulty. 

Measures of Cost-Effectiveness 

A final issue of concern is the cost of collecting data for each of the three 
methods. The cost figures included in this analysis generally include only the 
time the interviewers spent searching for and interviewing participants. For the 
telephone interview subsample, " interviewers were required to spend time attempting 
to locate every member of. the sample. This time was spent utilizing the contact * ' 
methods outlined above. For those individuals who were located, time was al$o 
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Table B-10: Check of Consistency jmd Completeness, 
, of Follow-Up Questionnaire 
(Percept Incorrect) / 



Choice No. 1 

Choice No. 2 

Choice No. 3 

Chpice Ho. 4 

Skip No. 1 
Skip No. 2 
Skip No.. 3 
Skip No. 4 



Telephone . 

6/36 (17%) 

7/87 (8%) 

5/26 (19%) 

* * * • 

3/51 (6%) 

2/36 (5%) 
6/45 (13%) 
1/51 (2%) 
0/51 (0%^* 



Personal Visifr ; 

\ 1 "5* 

6/38 (16%) • 

5/78 (6%) 

3/26 (11%), k l 

3/52 (6%) 



1/38 (3%) 

7/49 (14%) 

2£52 (4%) -5 

0/52 (P%) 



frail 

0/20 (0%) 
3/3# (9%) . 
5/6* (83%) 
fl/25.-<0%) 

•1/20 .(5%) 
V4/11 (36%); 
;^/25 (12%) 
0/25 (0%) 



'4 
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spent administering the interview. For the personal visit interview subsample, 
interviewers were also required to spend time attempting to locate every member of 
the sample. However, for those sample members who were located r time was spent 
initially arranging an interview time and date, traveling to the interview, and 
finally, administering the interview. For the mail survey subsample, time was 
spent preparing the twcrmailings and coding the responses contained on each ques- 
tionnaire as it was returned. 

These cost figures do not include two areas of cost. First, the figures^So 
not include time spent in supervising by the survey manager, or in checking and 
editing completed questionnaires by the interviewers. Second, each completed in- 
terview must be keypunched and entered into a computer ^system, and the resulting 
data must be edited and cleaned. These costs of the data processing related 
activities are not included in the cost figures as they do not vary by contact 
method, and therefore do not affect the relative cost of each method. 

Table B-ll presents the cost figures for 'each of the* contact methods. It can • 
be seen -that in terms of total cost, the pers9nal visit interview method is the 
most costly, the mail survey method is the second most costly,' and the telephone 
interview method is the least costly. Because the mail survey t is assumed to con- 
tact all the participants for whom questionnaires are not returned as undeliverable 
by the postal service, the mail survey yields the highest contact rate and the 
lowest cost per contact. However, given the relatively small number of completed 
interviews which results from the mail survey, it yields only the second lowest" , 
cdst per completed interview. Although it might^e anticipated that the mail sur- 
vey would be less costly because no interviewer costs are ^incurred, the substantial 
investment in time required to complete trie mailing and .code "the, responses is not 
offset by a higher number of completed and returned questionnaires. 

I 




Table B-U: Total 'Cost, Cost Per Contact and C6st Per dompleted 
Interview by Each Contact Method and in Total* 



Telephone Personal Visit Mail Survey Total 

Total Cost $127-61 $715.90 $197.74 $1,041.25 

Number of Participants Jf 

Contacted * 71 '61 125 257 

Number of Interviews 

Con*leted . 60 44 30 134 < 

Cost/Contact:' $ 1.30 $ 11.73 $ ^1.58 $ 4.05 

Cost/Completed Interview $ 2.13 $ 16.27 9 $ 6.59 • $ 7.77 



* These results are bas'ed 6nly on data collected at the Region VI site. 
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The table also shows that the telephone interview method is the least costly 
method and that the personal visit method is substantially more expensive. This 
large difference between the cost of the two methods is due primarily to ttie time 
spent traveling as part of the personal visit method. The personal visit method 
also 'involves two contacts vii?B the participant. The first contact i$ the initial m 
telephone tontact to arrange a time for the interview and the second contact is the 
in-person contact to administer the interview. This result points to the relative 
efficiency of the telephone interview method as a means of locating, contacting, and 
interviewing participants. 



"""The table shows that even in the case of the education characteristics, the deviation 
is small. The Chi-Square test indicates rejection of the null hypothesis only at the 
10% level. 

^ese equations are estimated by employing the ordinary least squares (OLS) method. 
It is well known that when the dependent variable is a zero-one dichotomous variable, 
the 'coefficient estimates are biased when OLS is used. However, as an approximation 
OLS has been employed to obtain the estimates which have been presented here: See: 
Pindyck, Robert S. and Daniel L. Rubinfe'lff; Econometric Models and Economic Forcasts; 
(McGraw Hill Co., New -York, 1976) Chapter 8, pp. .237-264. 
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APPENDIX C 

•j^Lephone/Persofral Visit Follow-Up Questionnaire 



/ 



\ 



?fll*Z 'SPONSOR FOLLOW-';? PHCJECT 
7X7LS XX 

?;urrxcx?ANT follow^? ccssTicsNaias 



Svery box should be filled in. If a question is not answered do not Itave she boxes blame. 
Instead, fill then ir. with one of the following codes unless different siss mg value cedes 
are specified : „ * 

» 7 - If the respondent refuses to answer 

w 8 - if the question ts r.ot applicable 

9 - if the respondent does not fcnow 



Tersmee's Masse: 
Telephone: 



Address: 



Other Phone/Contact Xnforaation:_ 



Card Muoher: 



X.O. Musoer: 



1 2 



0 I 1 



Date 



Interviewer 



?rioe Sponsor 1.0. Uumber: 
Dace Interview Completed: 



18 



m 



17/ 



23 



(no.) (day) (yr.) 



How Contacted: <Code only the last method/strategy used to contact respondent.) 



1 • Priae Sponsor Intake Information 

2 - mployer/School/Other Prime Sponsor Records 

3 • Post Office Correction/Toilow-Up Mailings 



"4 « Friends or Relatives 

5 • Telephone Dire ctory/Di rectories 

6 « Other „ 



24 
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"Hallo, ay name is » » I am railing .rom 

Wo ere =*rformtr.g an evaluation c i she CETA program in ^nx;h 



you participated about six sontns ago. Ifou nav* proeaci/ rscaivec s lector frota 
us recently^ telling aeoue this soucy ■ I would like to asx you some questions 
about what you thought of tr.e CETA program and wr.at you nave been, ^omc sine* you 
left the program. Oo ycu have time now?" 
t • * 

Scenes i: prsraoEyrs vcw or the psccsam 

"The first series of cjxestions has -o do w^h your views of the CETA program. H 

«25 26 



What was your MAIM reason for enrolling in =ne CETA program? 

01 • To Get a Job 05 ■ H.5. Diploma/~GEC 

02 • To Get a CETA Job 06 • Basic Education Services 

03 » To Leern A Skill or Trade 0? - To Learr. rngli3h 

04 - Heeded >toney 38 ■ Other^ 



2. Ail things considered, how would you rat* she CETA program? 
(Read choices to respondent) 

1 - Excellent Z ^ Zacd 3 - 4 - ?oor 

What was the 3EST thing about tne CETA program? 



4. What was the WORST thing about the CETA program? 



27 

□ 



5 . How would ''ycu improve the CETA program? 



t 
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1980 



5 M T W T F S 
APSIt 

1 2 3 4 5 

6 7 3 9 10 11 12 
13 14 15 16 17 13 19 
20 21 22 23 24 25 26 
27 28 29 30 

/MAY 

1 2 3 

4 5 6 7 3 9 10 

11 12 13 14 15 16 17 
13 19 20 21 22 23 24 

25 2S 2f 23,29 30 31 
JUNE 

1 2 3 4 5 3 7 
3 9 10 11 12 13 14 

15 16 17 13 19 20 21 

22 23 24 25 25 27 28 

23 30 * 
JULY 

1 ? 3 4 5 
• 6 7 ,8 9 10 11 12 

13 14 15 16 1? 18 19 

20 21 22 23 24 25 25 
27 23 29-30,31 
AUQUST ( ; 12 

3 4 5 6 7 8 9 
10 11 12 13 14 15 16 
17,18 19 20 21 22 23 

24 25 2527 28 29 30 
31 

SEPTEMBER 

1 2 3.4 

7 8 9 10 11 

14 15 IS 17 18 

21 22 23 24 25 2S 
23 29 30 
OCTO?ca . 

1 2 3 

5 S 7 8 9 10 

12 13 14 15 16 17 18 
19 2a 21 22 23 24 25 

26 2Z 23 29 30 31 
NOVEMBER 

2 3 4 5 6 7 
9 10 11 12 13 14 

16 17 13 19 20'21 
23 24 25 26 27 23 29 
30 

DECEMBER - 

1 2 3 4 5 6 
7 8 9 10 11 12 13 
14 15 16 17 15 19 20 
21 22 23 2X 25 26 27 
23 29 30 31 



4 
11 



1 
3 

15 
22 



1981 



S M T W 


T F 


JANUARY 






1 2 3 


4 5 6 7 


8 9 10 


11 12 13 14 


15 16 17 


18 19 20 21 


22 23 24 


25 25.27 23 


29 30 31 


FEBRUARY 




12 3 4 


5 6 7 


8 9 10 11 


12 13 14 


15 16 17 13 


19 20 21 


22 23 24 25 


25 27 23 


MARCH 




12 3 4 


5 6 7 


Z 9 10 ii 


12 13 14* 


IS 15 17 13 


19 20 21 


22 23 24 23 


23 27 21 


29 30 3t 



3£rCR£ STARTING i (Circle the date *^e. tsrainet «nded the CETA pcogr&n and today's 
dace on the, calendar at left. Work backyards on the calendar 
covering the entire time period as you fill out the chart below } 



SECTION II: SUMMARY POST 



-PROGRAM EXPSRIE* 



ENCZ 



"Now I would like to fir.d out what you have been doing sn 
is betveeq (date encled last program) 



f 



ending your CETA program, that 
and now. I would like to 



start by asking you about what you are doing now. and tnen ask you a series of questions 
about your activities during the past few oonths. 



6A. 



"Which of the following activities are you doing currently? 1 * (Read 5 Activities to 
Respondent) 



Activity Code Activities f 

^ 1 • Working 

2 » Attending School 

3 ■ Attending a Training Program. 

*% 

^4 * Serving in the Arced Forces ■ 

5 * None of These 



! GO TCJ QUESTION 7 (In Chart) 



-GO TO QUESTION 11 (In Chart) 



63. 



"Before that, which of the following activities were you doing?" 
to Respondent) 



(Read S Activities 



Activity Coda 

1 
2 
3 

A 

5 



Activities 

Working - 

Attending School 

Attending A Training Program- 

Serving in the Arced forces • 
None of These „ 



;C0 TO QUESTION 7 (in Chart) 



TO QUESTION 11 (In Chart) 



( REPEAT QUESTlOttS 63. THROUGH 1£, AS APPLICABLE . UNTIL THE ENTIRE POST-? ROC ?AM PERIOD 
IS ACCOUNTED ?OR> 



Activity 
Code 


7. What is the 1 
Company/ 
School 
name? 


8. Where is it 
located? 


(If Working) 
9. What does 
this 

company 

do? 


(If Working) 
10. What is/ 
was your 
, job title? 


-l.When 
did 
you 
begin? 


12. When 
did 
you 
stop? 
(GO TO 
Q. 6B) 


* 

of 

Weeks 


1) 












PRESENT 




2) 
















3) 
















4) 






\ 


s 








5) 
















6) 
















7) ' 
















8) 








i 









13. Are any of these jobs "*CETA jobs M ? 

< 

If Yes: Which ones? (Circle row number) 



t** WORKING AT TI!!E ENDED PROGRAM: GO TO SECTION IV 

NOT WORKING AT TI>2 ENDED PROGRAM : GO TO SECTION £51 
JOB SINCE LEAVING CETA: GO TO SECTION VI 
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szcrrcK in-. ::qt wqrxing prior to the first Jcs after cet\ 

(Fot RasoondAnts Who Wtre Mot Wording Ac The Tine The Program Ended) 



"You* and chAt b«tv««n iAaving CSTA And ^ 

I would like co *ik a ftw qut scions *xout ztaz 



you did noc 

waak cise . " 



15. 



16. 



Old you look for work during Avery vaaJc of chac ci»« ?enod? I 

ir res ; 3c ro section rv 

IT MO i GO TO QUESTION 15 

How aany wAeks did you look for work during chac ciae? ¥ 

(If 100% of ChA tlSA) : GO TO SECTION IV 

(If Iass Chan 100% of the tiae) : 00 TO QUESTICN 15 



Yes 2 » :io 



During the tiae whAn you did noc loox for wor*> wnat *as t.-.e :*AZV, 



01 - 3tli«ved No Work was Avail ad 1a f 

In Lint of 'Work or Ar^A 

02 » Couldn't rind Any Work 

03 - Old Noc Wane co work 

04 * LackAd NACASsary Schooling, 

, Training. Skill or Exptritnct 

05 ■ EaployATi Thougnt Too Young or 

Too Old 



c 

:6 » 111 Health, Physical Oisabilic 

07 - Couldn't Arrange 'child Car a 

08 ■ Prtgnancy 

09 ■ Other Personel Handicap in 

rinding a Job 

10 - Other 



28 

»□ 



IS 



16 



29 


30 




1 


31 


32 




i 



SECTION IV: INFORMATION' ON THE FIRST OR ONLY ZZtPLOYER AFTE?. CSTA 

"Now I Would UkA To Get MorA Information About The Job YoU Ha Id (Are Holding; At 
as a ■ " 



(CoopAny Najoa) ^Job Titlt) 

17. How Did You Find Out About This Job? 



01 - CETA AgAncy 

02 • EmploytaAnt OfficA/Job ServicA 

03 ■ Private Employment AgAncy 

04 " ContactAd EmployAr Directly 



05 ■ "rC^nds or StUtivts 

06 ■ AnswerAd NswspapAr Ad. 

07 - Other 



13. WhAt was your stArting hourly wagA ratA? 

19. WhAt was (is) your final (currAnt) hourly wagA ratA? 

20. How oany *hours did (do) you usually work ft w#Ak? 

s 

21. (If no longAr working for this AaployAr) 
Why did you leave this job? 



35 



13 S 
19 S 



33 34 



0.7 



CD 



'33 



42 



01 • Unsatisfactory work Conditions or * 
w ArrangADents (Hours, Ate.) 

02 ■ Found Hightr Paying Job 
Q3 - RAtumAd to School 

04 - Problems with Child Cars 

05 ■ Prsgnancy 

06 - 111 HAAlch, Physical Disability 



07 - ReachAd Retireaenc Act 

08 ■ Laid Off Du.a to Poor 

« Business Conditions 

09 ■ Tenporary or Seasonal 

Job End Ad 

10 - Dismisssd/Firtd 

11 - Oth Ar , 



20 



43 44 



45 46 



21 



176 



(J 



period iNfcayATioM om cumest/last z.hployir :? -ypas ?ha:i cne emsl^ye?. after ce?a 

"Vow I vould like J to ?et more indorsation About tne :ob you're noldin? iow (last ;oo 
vou held) *c as * . , • 



22. 



{ Company ^t) 
what was your starting nourly wage rate? 



^Job Title) 



22 S 

23 5 



50 



23. What is (was) your current (final) hourly wage rite? 

24. How many hours clo (did) you usually work per w««<? 

23. (Lf no longer working for this employer): Why did you leave this :ob? 



r Dl 



"srss 



24 ; 



25 



01 • Unsatisfactory work conditions 

Arrangments (Hours, etc. ) 

02 • found Higher Paying Job 

03 * Returned To School 

04 - Problems with Child Cars 

05 • ?r agnatic? 



06 - 111 Health* Physical Disability 

07 • Reached Retirement Age 

08 - Laid Off Out to Poor Business 

Conditions * 

09 ■ temporary or Seasonal Joo Ended 

10 • sLsaisssd/Fmud 

11 - Cdver 




SECTZOWV/I: FOR RESPONDENTS WHO ARE ?ICT CtTRRENTLY TORXyG 



saidVyo^isOava not worked since 
itw questions about this 



I would like to ask you 



wetk period. 



26. 



27. 



23. 



2 



Mo 



Have you looked for work? 1 fa Yas 

many of the weeks have you actually looked for *orK? 



H<L 



2$ . 

60 



59 



27 



Ars you currently looking for work?* 1 • Yes * 2 • Mo 

( l\f Yes to 23; Ask Questions 29 and 30, Then Ask Question 32 and 33) 

(List two ways) 



'aaU 



29. How are you looking for a Job? 

01 • CSTA Agency 05 - Friends or "Relatives 

-02 • Employment Off ice/ Job Service 06 • Answering Newspaper Ads 
V3 * Private Employment Agency 07 ■ Other ^ 
04 • Contacting Employers Directly 

\Q. *ha^ do you think is the MAIN reason yo.u have not been able to find work? 



29! 

»l 

30 



63 64 



65 66 



05" » Lack of Tools, Licenses 
C6 • 'Language Problems 

07 « Police Racord 

08 * Transportation Problems 

09 • Other 



31/ 



01 - No Jobs Available 

02 - Employers think too young 

or too old 

03 • Lack of Skills, Schooling 

or Experience 

04 * Lack of References * 



(If Mo to 28: Ask Questions 31 through 33) 

What are the two MAIN reasons you are not looking for work? 



69 70 



06 - 111 health, physical disability 

07 - Can't arrange child care 

08 * Pregnancy 

09 • Other personal handicap in finding 
a 30b 

J.0 » Other * 



32. 



33. 



01 - Believes no work is availably 

in line of work or area . 

02 • Can't Find any work k , 

03 * Do not want to work A 

04 - Lack necessary schooling, 

skills or experience 

05 • Employers think too young ,* * 

or too old . 

Are You available for work right now? 1*- Yes 2 - Mo 

j 

Are shere any other things you 'would like to tall us about the CSTA program? 



31 



31 



1 


.1 


■71 


72 


! 



32 



73 

□ 



THE INTERVIEW IS COMPLETE. PLEASE* STOP, 
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SCMHA^Y SECTION 



tThe purpose of this section it to 5ummarire tne ir.ftrmation ?atr.«rad m tr.a Tacle 
in 3*cticn II,) 



Card Number: 
:.D, Nun£ar: 

Part Z: Miration of Labor 



C. 



^kZK*Z ActlVV 



,ies ih tne Post-rrocraa Paricd 



A. Number of Weeks 3ecveen Oat* Ended Program arxl Foilcv-tfp Interview: 
3, Number of Weeks Spent Working on Sacn Joo: 



1. First Job: 

2, Second -ooj . 
3 T Third Job: 

4. Fourth Job: 

5, Current/Last Job: 
(Add 3.1 througn 3,5) 



Total 



12 
14 
16 
18 
20 



Number of Weeks in School Whan Not Working; 
Number of Weeks In Training Whan Not Worjcing: 
:;uob«r of We|fcs Spant in Armed Forces i 

Number of leaks When Not working, Not m Scnool, Not in a 
Training Program and Not in Armed forces: 



• 3. 
4. 



3etvear. Er.d of Program 
and First Job: 

Before Second Job: 

3efore Third Job: I 

Before Fourth Job: 

Jo: 



5. Before Current/Last Job: 

6. After jCurrent/Last or Only Job: 



7. Total Weeks Noje Working , Not in 
School, in a Training Program 
or Not in the Armed Forces: 



30 
32 
34 
36 
38 
40 



(Add F.l through F.6) 



Total 



(Please Note: The answers in Boxes 3. C. D# Z and F should add up 
to be approximately equaj^to the answer found in Box A, ? least 
sea Instruction Manual for details.) ^. 










22 


3 














24 


C 














26 


D 














28 


£ 









I 42 
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PlCrt II: Indicators eg ?rocraa Cutccnes 



1 



5. On The' Date of Termination, 'Did the Xescondent Have A Job? . , 
I • *es 2 ■ Ho % 

H. What xss the Total Nunber of Jobs aetFondent Held Sine a Uiving CTTA? 

I. Were Any of the Joat Identified at ?SE Jobs? * 

I ■ First 2 ■ Second 3 ■ Third 4 " Fourth 5 » Current/I^st 6 • :io 

J. Is the Respondent Currently Saployed? i - Yes , 2 --No ? 

X, Is the Respondent Currently Attending School or Another Training Progran? 
I » Yes ' 2 - :<o » . 

/ 

?art III: Industry and Occupation Codes 

Sl&t 000 * Mot Aqpiicahle/Inforaation Mitt 1x19 

DOT* OO0.00O-0CO ■ .'lot Applicable/Inf onation Missing 

L/ Information for First Job 



I. industry 



_SIC< 
4 



2 . Occupation^ 



UOTi 



M. Information for Current/Last Job 
~* 1. Industry S*£» 



2. Occupation^ 



DOT: 



52 



64 



M2 





/ 











M 



C»rd Suab«r! 



1.9. Nusbar: 



It 



9 

■ERIC 
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Part rV: Oacea 3«<;an and SLidtd Jobs 
:i. First Joo; 1. Date Began: 

2. Dace Endtd: 
O. Second Jcb: 1. Date 3egan: 

2. Cat* Ended: 
P. Tfeird.Job: I. Dace 3egan: 



2. Dace Ended: 



fpurtn "Job : 1 . Data -#egan * 



2. Dace Ended: 
Current/last Job: I. 



Dace Began: 



Ml 

01 
02 



2. Dace Ended: 



?2 
Ql 
Q2 
Rl 
R2 



19 



3T 



20 



26 



32 



~ 38 



44 



180 



^ -* APPENDIX D 

Mail Survey Follow-Up Questionnaire 



■ i. 



. ERIC 
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Dear CETA Participant: 

» • 
In order to' improve the overall quality of CETA training, we are. conducting 
a follow-up ev^luation^bf the program in which you participated. You may have 
received a letter a fe&r weeks ago informing you of this, study, Your views of tfte % 
• progtam and some general information about your employment situation will be very 
helpful* You can be assured that any information which you provide on the question- 
naire wiiqj be 'held '"in "strictest of confidence., 1% * 

Attached you will find a brief questionnaire. Not all sections of it will 
apply to you. Instructions 'are, provided throughout the questionnaire where necessary 
Only five or ten minutes of your time should be required to complete it. 

Thank you very m^ch for your cooperation in this study. . * 



f \ 



182 



ceta follow-c? ^csstickna:?* 

First, we would iiki tc ask you a Jew questions aoout your opinion of the 
prograa. ' , 

1. What w*s your MAX* reason for enrolling in tr.e CTTA program? Please circit 
one number. 

i. To Get a Job $. * To Get*a Sign Scnool Diploae or a 'JED 

2«. To Gee a CETA Job 5. To Gee Basic Education Services 

3. To Learn a Skill or Trad*' 7 . To Learn English 

4. Needed Money 3. Other Reasons that are riot Listed 

Hart. Please List Other Reasons 



2. All things considered, 'how would you rate the CETA program? Please circle 
on* number. > 

1 . excellent 2 . 3ood 3 . Fair 4 . Poor 

3. What was the 3EST thing about the CETA program? Pitas* writ* wnat you 
thin* is the best thing eoout the program on the blank line. 

_^ : '— 

*4. What was the WORST* thing about tne CTTA prog ran? Please write what ycu 
cbinx is the worst' tning about tr.e prograa on the blank line. 



How would you inprovV the CETA prograa? Please write how you would improve 

the prograa on the olank line. 

• # 



- x 
Now, we would like to find out what you've been doing during the last stveral aonths. 
We are interested in finding out about the tiae you cay have spentf in five different ^ 
types oi activities . These are: 

i. Working 
, 2. Attending School t % 

3. Participating in a Training Prograa 

4. Servirig in the Araed Forces * 

5. Sot Working (also not la school, a training prograa or 

the anted forces) 

we would like you to think about -which of these activities you were involved with 
since you«stopped participating in the CSTA prograa. Then, we would like you to 
fill out the calendar on page 3. The calendar is designed to record the tiae you 
aay have spent in each of the above activities. But, before you go ahead and fill 
out the calendar on page 3, please read the following example which nay help you. 

Suppose Pat Greene stopped participated in the CSTA prograa on July 17, 1980 
and is filling out the -calendar on February 5, 1981. Suppose Pat has held one job 
and attended school since leaving the CSTA prograa. The :ob began on August 1, 1980 
and ended on November 3, 19^0. And, Pat beqan attending school on Oeceaber 3, 1980 
j^tt is stiil v attending school. 

Step It On page. 2, An "X" has be*e R aar*e< on the calendar on July 17, 1980, the day 
'"^ — Pat left the prograa, and on February 5* If 81, ^he day Pat is filling out 
^ the calendar. * % - 

Stto 2- On page 2, Pat has also circled and labeled the days on the Calendar when 

' ' h# WA , WORKING, NOT HOPJCXMC and ATTEND I?EG SCHOOL . Froa Julfr 17, 1980 tp 4 

August T, 1980, Pat was MOT WORKING . Front- August 2, 1980 to November 5, . 
1980 , Pat was WORKING. Frca November 6, 1980 to December 7, 1980, Pat 
was NOT WORKING . Finally, froa December 8, 198^ to February S, 1981, Pat 
was ATTENDING SCHOOL . V ' 
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Sow vou she aid thin* about vnicr. of the activities listed above you heve =e«r. 
involved in since Uavtag tne CSTA prograe. Xour experiences =ay not have been the 
sace as tne exasple presented here. For instance,* if you have b*en involved m two 
activities et once, "such- as working and attending scaooi, please be sure to aark it" 
on the calendar. Please fill out the calendar on the next page using 5tep,l and 
Step 2 listed above. c 



a. 



TV ? 5 



3 



21 '23. 



3 |2S j jo 



tr 



IX 

IT 



25 




7T 



20 2J 



!25 



7J 



A. ]3 



21 3" 



£1 -3 1^ 

I I 



r 



h 



2* 



2f !25 2^ 



-J 



/v 



/a 



/5 



a# * 



27 



2? 



iz. 



UB in lift ft-4 x 'i 



as 



/T 

ST 



3 JJ 



3 



I 



/2 




is 



5T 



2^~xi 



a. 



4/ 



4a s/ 

is 



5® 



7T" 



/; 



f 



03 \2t* 



TT 



30 



2T" 



/A. i?3 



Iff 



Tcl » W i Th 



23 



n 



i5at 



24 



31 



IS 



05 



a. 

It 



TTTi 



3a c 

"3o / |A |y ;V ^""j*" 



TT" 



15 



75 



7TT « 



I 



at 



7? W 



a » ^ ^ 
5rt\ * *i 7 ? 



27 2# 



42. 



/O t 



n 



25 



2f 



3| 





A. 


5 








7 


« — 


4 — 






■Mr- 1 






15 


x» 


/> 


/7 




20 


2J 


*X 


23 


X? 


25 


26 


27 


2T 


/ 


2. 




y 


5 




\ 


X 


f 




// 


/A 




/v 


/* 




/7 






2o 


2/ 


22. 


23 


2«/ 


as 


36 


27 


2X 


2t 


?° 




i 
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<5 


'At 




w 




• 








/ 


X 


J 




5 


y 


7 




<? 
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II 
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15 


/* 


/7 




7? 

• 


v> 


21 


33. 


A3 


ai' 


05 




XI 


at 
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5 


* 


7 


* 


T 


/D 


// 


/a. 


B 
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IS 


/? 
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ai. 
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? 
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at 


*7 
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S3 


30 
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A. 


5 


H 


5 


'o 


7 
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/» 

r 

f 


a 
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(5 


/y 
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/f 
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a/ 


as. 




*V 
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to 
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to 
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IS 
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a/ 
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3 


</ 


5 


& 
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/o 
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74 
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AS 


as 

t 
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to 
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hi 
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Tu. ' 


W 


T7r 


F I 


5af 




b 
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// 
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W 


IS 


lb 


n 


/J 


/? 


ao 


aj 
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a^ 
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as 
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3 
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/O 
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23 
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/7 


IT 


^? 


30 


a/ 
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sow we would like to ask you some cniestions aoout vhat you've been doir.c since you 
left the^CSTA ?ro<jrafl.i Please circle those answers wnicr. 3eem sest. i:ot all tnese 
questioftsvtll apply to* you, so please follow the instructions beneatn eacr. question . ^ 

1. If you have not wtfrked since you left the CETA program, please turn to Question 
34 and continue completing this form. If you nave worked, please continue with 
Question, 2. 



2. On the DAY you stopped attending the CZ7A program did you have a ;oo? 
Yes, Go to Question 6 and continue. No, Answer Question 3 and continue*. 

3. Between leaving the CZTA program and beginning your first 30b, did you look for 
working during EVERY WE£K of tnat tine period? • 

Yea. Go to Question 6 and continue. Mo. Answer Question 4 and continue. 

4. 'During the tine between leaving the CZ7A program and beginning your first ;ob, 
how many weeks did you look for worx? Please write the number of weeks m the 
box. It may b/ useful to look at the calendar wnich you have already filled 



During tnis time period^ vfien you did. not look for work, wnat were t; 
reasons for not looking? Please circle the two reasons. 



«.e two MAIS 



4. 



I believed no jobs were available 

in my line of work or ay area of 

interest. * 

I couldn't find any kind of work. 

I didn't want to work- 

I lacked the necessary schooling, 

training, skill or experience. 

Employers thought that I was 

too young or too old. 



6. I had poor neaith or a physical 
disability. 

7. I couldn't arrange child care » 
3. I was pregnant. 

9. I had other personal problems m 

finding a ;ob. 
10. Other reasons that are not lisced 
nere. Please list these other 
reasons , 



5 ^ Have you held more than one 30b since leaving the CSTA program? 

Yes. Answer Questions. 7-16 and continue. 
So. Turn to Question 16 and continue. 



How deny Jobs have you held since leaving the ,CSTA program? write Jhe 

number of jobs in the box. 



1 9. 



3, How did you find out about your first job which you held after leaving the 
program?v Please circle one number. 



Local CSTA Office 
State Employment Office 

(Unemployment office) 
A Private Employment Agency 
Contacting the Employer Directly 



5. Friends i Relatives 
. 6. Answering a newspaper ad 
7. Other ^ays not listed here* 
Please List 



What is the name of the first company 
you worked for after leaving the prograa? 
Please vr,ite the name of the company on the 
blank line. 
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10. What dots this company do? Please writ* 
vnat c.-it company dees on ^ne blanx line. 



11. *What was your job , title? Please write 

your ;ob title on the. blank line. . 

12. <Wh*c was you starting hourly wags rats? Please writs you -/age rata in tr.e box. 

If you wara paid by the week and do not know your starting %ur' ■ -act race, 
please writs your starting weekly salary before any deductions *n tns box. 



13. 



What was your final hourly -wage rata? ?laass write your wage rate m the> box. , 
If^vour final hourly waga rata was the same as your starting hourly wage rats, 
iaave the box empty and answer questions 14 througn 16 If you ware paid by the 
waeJc and do not know your final hourly wage rata, please write your final weekly 
salary * before any deductions in the box. 



14." How oany hours psr waek did you usually wor* on tnis }ob? Pleas* put the numcer 
of hours in the box. 



15. 



Why did you laav* this ;oo? Please circle the MAIN reason. 



1. The working conditions 

wars unsatisfactory. 

2. I found a higher paying 

job, - 

3. I returned to school. * 

4. I had problems with child 

care. 

5. I was pregnant. 



6. I had poor health or a physical disabrli 

7. I reacned retirement age. 

3. I was laid cff due to poor business cone" 

9. The :ob was only seasonal cr temporary. 

10. 1 was dismissed or fired. 

11. Otijer reasons that are not listed here. 

Please list other reasons. 





16. Are 


you currently employed? 








Yes 

Ho. 


Answer Questions 17 through 23 
Turn to Question 24 through 32 


and 41. 

and continue. , 

















How did you find out about your current job? Please circle a number. 



17. 



13. 



19. 



6. 
7. 



1. Local CSTA Office , 

2. State* Employment Office 

(Unemployment Office) 

3. A private employment agency 

4. Contacting the' employer 

directly 



What is the name of the conpany you are 
working for? Please write the name of 
the ccepany on the blank line. 

What does this company do? Please write 
what the company does on the blank line. 



Friends 6 Relatives 
Answering a Newspaper Ad 
Other ways * not listed here 
Please list ' 
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20. 



21 , 



What is your 300 -it!*? ?Uase writ* 
your job title on the blank line. 



What was '/oar starting hourly wage rate? Pleas* writ* your -rage race in t; 
Xf you axe paid by the weex and do not kr.ow your starting aouriy wag* rat 
olees* write your starting weekly salary oefore ar.y deductions in tne box 



22. 



23. 



vrite^our wege rate in the box. 
♦y^our Starting hourly wage rate* 
If you ere paid by the week 



What is your current hourly rate? Please 

If your current hourly wage, rat** is 1 the sane 
leava the box eapty and answer questioVs/23 an^ 
and do not know ycur current hourly wageTSte, please write your current Weekly 
salary/ before any deductions/ in the box. 



How aar.y hours per/ week do you usually work on this 
of hours In the box. ~ 



:ob? 



? lease out tne number 

y 



•*** Please Turn to Question 41 ••■ 



24. 



Hoy did you find out about your last :ob wmch you held after leaving the 
-ran? Please circle one nuaber. 



1. Local CSTA Office 

2. Stat* Employment Office 

(Unemployment Office) 

3. A Private Employment Agency 

4. Contacting the Employer Directly 



Friends & Relatives 
Answering a Newspaper Ad 
Other ways not listed here 
Please list _ 



25. .How .we would like to get some information about your last employer. 

What is the name of the last company 

you worked for after leaving the program? 

Please write the name of the company * 

on the blank line* * 

26. What does this company do? Please 

write what the company does on the 4 4 

blank line. ■ > . 

27. What was your job title? Please 

write your Job title on the blank v/ , 
line. _ . 



28. What was /our starting hourly wage rate* Please write your wage rate in the box. 
If you are paid by the week and do not know your starting hourly wage rate, 
-please writ* your starting weekly salary before any deductions in the box. 
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23. *.*hat was your final nouri/ wage rata?.. Please -n::.te /our *act rata in -he box. 
If your final hourly ^aga rata was we jam© as /cur startir.3 r.ourly vace rate, 
leave the box eapty and inswer questions 30 tnroucn ZZ and continue. If you 
are paid oy the week and do not cow your final nourl^ vaga rate » please write 
your final weekly salary, oefore any deductions in vie box. 



30. How many hours.per weak did you usualiyf work on this :ob? Please put the number 
of hours m the* *?ox. 



31. Wiy did you leave this ;ob? ?ltast circle 

1. Working conditions vara 

unsatisfactory. 

2. I found e higher v ?ayir.g 30b* 

3. I returned to school. 

4. I had problems with cm id care. 

5. X was pregnant. 



the MAIM reason. 

6. I had poor health qr a physical 

disability. 

7. I reacned retirement ace. 

8. I v«s leid off due to poor business. 

9. The :ob was only seasonal or 
* temporary . 

10. I was dismissed or fired. • 

11. Other reesons that are not listed 

here. Please list other reasons 



32. 3ince having finished working, fcr your last employer, have you ioo/«ed for 
work? 

Yes. Answer Questions 33 and continue. No. Answer Questions 39, 40^ and 41. 

13. Stnca having finished working for your last employer, how many weeks did you 
look* for work? Please write the 'number of weeks in the box.** It may be useful 
to look at the calendar which you nave already filled out. 



•••• ?laese Turn to Question 36 **** 



34. Since leaving the CSTA program have you looked for work? 

Yes. Answer Question 35 and continue. So. Answer Questions 39 , 40 and 41. 

35. Since leaving the CSTA program, how many weeks did yeu look for work? Please 
write the number of weak* in the box. It may be useful to look at the ceiendar 
which you have already filled out. 



36. Are you currently looking for work? 

Yts. Answer Questions 37, 38 and 41. :io. Answer Questions 39, 40 and 41. 



■ J 
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38. 



you looking for a ;ob? Please circle two. ways. 



How are i 

Local CTTA Office 

2. State Employment Office 

<Vnemplcynant Office) 

3. X Private Employment Agency 

4. Contacting tne Employer Directly 

What do you think is tha >UIM reason you have not been sole to find wor<? 
Please circle the MAIN raason. 



5. Friends & Relatives 

6, Answering a Newspeper Ad 
r . other-w^ys not used hara 

* ?iaasa list 



1. Mo jobs ara available.. 

2. Employers think I am too young or 

too old. 

3. Z lack tha necessary schooling, 

training, skill or axparianca. 

4. I lack tha required rsfsrences. 



5. I lack the nacassary tools or 

licenses 

6. I have a language problem. , 

7. I nave a police record. 

3. Z have transportation problems. 

9. Other reesons that are not listed 
here. Please list these other 
reesons 



***« please Go TO Question 41 



What Are the two MAIN reasons you ar« not looking for work? Please circle the 
two MAIN reasons. 



39. 



40. 



1. 


I believe no Jobs are available 


6. 




in ay line of work or my area of 


7. 




interest. 


2. 


X can't find any kind of work. 


3. 


3. 


I don't want to work. 


9, 


4, 


I lack the necessary schooling* 






training, skill or. expedience. # 




5. 


Employers think that I ar too 
young or too old. 





I have poor health or a physical 
disability. 

I can't arrange child care. 
Z am pregnant. 

I have other personal problems m< 
finding a 30b. 

Other reasons that are not listed 
here. Please list those other 
re**oHr^ 



Are you available for work right now? 

Yes. Answer Question 41. No. Answer Question 41. 



41 The interview is complete. Thank you very much for your help. If there are 
' any other things you would like to tell as about the CSTA program, please use 
the space below for your comments. 



Please Stop. Please return the questionnaire to us by using the 
envelope which has been supplied to you. 
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APPENDIX ,E 

Region One Follow-Up Evaluation Project Follow-Up Questionnaire 

* # 



zszzzu cne roi&cw-c? evau;at:cn pkcjec? 

PREPARED 3Y: 



PCLICV EVALUATION 3IVISICN 
DEPXRT1SKT OF riANPOWZR DEVELOPMENT 
COMMONWEALTH OF MASSACHUSETTS 



tN CONJUNCTION tfTTH: STGXON CNt FCLLCW-tf? EVALUATION PROJECT 

Every box should be filiad in- a question is not answered do not lave the * 
b5Ss blank, tuciid, .'iUduam with ont of tht islloviag codes unless d^ftrgnt 
missing value codti art specified! 

7 - If the respondent refuses to "answer 
3 - i* the c;jestion is not applicable 
9 - if the respondent dots not know. 



Teramee's Namei^ 
Address t 



Telpbone i 



Other Phone/Contact Information^ 



Date 










































i ■ 





Interview Status/Reason For Mo tnttrvitw 

01 - Completed -* 

02 • Mo Phone Mo. At Intaka 

03 • Incorrect Phone Ho. At Intaka 

04 • Oisconnactad Phona 

05 • Unpublished Phona No. 

06 • Moved* Address Unknown 

07 • Participant Rafuaed Interview 



How Contacted 

01 • Intaka Phone Number 

02 • Directory Assistance 

FoUoWtrp Contact Lettert * 

03 - Responded by Hail 

04 - Responded by Phone 

05 —Employer 

06 • Prime Sponsor Records 

07 ■ Other Social Service Agency 
01 - School 



Total Number Phone CaUs and Home Visits t 
Data interview Completed* 
Interviewer t.O.t . 

* 192 



09 » could Mot Locate 

10 - Language Problem 

11 - Death ■ 

12 • Participant could not 

13 • In Military 

14 • Oat of State Phone Mo. 

15 - Incarcerated 



09 • Military 

10 • Post Office Correction * 

11 - Relatives 

12 - rriends 

13 - TMIS 

14 » Program Operator Record 

15 - Other 



56 


57 




1 





5| 59, 



16 • NA or Not Contacted 



64 



60 . 6.1 

CD 



62 63 



\ □ i □ 



Card tfimber: 



SZCnCll I: KgSTCyDDrT'S VZER OF PROGRAM 

*The first series of questions has to do with your views of CSTA", 
i, What was your 3AIH reason for enrolling in the CSTA program? 



' 01 • A "CSTA Job" • 06 « Stipend 

02 - Help ffinding a job 07 - H.S. Oiploma/CT 

03 ■ A job 1 w 08 " *** ie Education Sarvices 

04 • K better job 09 ••Instruction in IngUsh 

05 - U4m a skill ' "Iw^freaaT" 7 " 




7?\ 2# y^t ^0 you think you would have done if you had not enrolled in ths 
CTTA pr jgram? w , , 



j ] 3, ' what was the BEST thing about £he,CETA .program? 



0 
O 



4, what was the WORST thing about tha CSTA program? 

■ . — & — 



5 All thing i considered, how would you rata tha CSTA program?/ 

1 • Excellent 2 - Good . 3 - Fair 4 • Poor • 

6. Do you have any, other feelings about tha CSTA program you would 
lJJca to express? 



V 



SECTION H: SUPPORTIVE SZityiCSS 

"low : would like to ask you a few questions about soma services ^;ou txay 
have racaivad whila in CTTA 4 . 

7. Whila in tha CSTA program did you receive any of tha following services 
from CSTA or any othar agency! 

a. Child Cara? 

b. Haalth Services? 1 • * M 2 " No 

c. Transportation Assistance? 

« d. Legal Services? 1 



* « 
vte to 7) 8. Did you racaiva enough « 

™ *° / , Uypei off aervicei received) 

a. Child Care? 

b. Health Services? 

c. Transportation Assistance? 1 • Yes 2 - No^ 



d. Legal Services? 



2 - 
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FOR RZSPON32ITS WO OtD HOT ?£ZZT«1 ALi FCUJl TYPES: 

9. Would you hav* h^«n abla to o/tt zor* out cf your CTTA prsgrap you had 
rtctivaH tothari strv&tts. that is: s 

a. Child cart? 

b, Htaltfc Strvtcti? - 1 ■ Ytfs 2 « :io 
* e. Transportation Ajsistanet? 

d. L*<?al Sarricas? 
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-row* r 'would lifea to find out what you havt b«tn doing n.-.ct «ndir.c your CSTA progran. 
that is b€Cwoft <d*ts tncUd procrao of tsrainaticrJ ii*d r.ow." 

10*. Art you currently working, »ttand*ng wheel, another -.raining program or doing t 
~ * southing «,!*•? 

tsr something nst) 

* :0b. Vhat is it that you art doing? (ask ^ssoicns 11 through 15 until the snrire 
'pcao-prograsTpariod is sccounrtd for) • 

■•Activity'cod.*: 1 - Snployd . 2 - School 3 - training Pro.grso 4 » southing Slat 



^Activity 3 
i 

r 

\ 


11. What mt tht 
Ccaoany/ " 
School nasa? 


l^Whsra is it 
located? 


{I* Hoiking) i 
iS.Shat it 

you* 

titla? 

* tV 1 
i 


4. Whan did 
you btgin? 

t 


IS. Khan did |16.3«fort that? 
you sccp?| vhat wars j 
' ! vou doing . 
j (that J*, 

working. j 
j school. j 
f are. 5 _! 




jx> 




i 


i 
1 




i 
i 


i 
i 
t 

i 


2) 




V. 

i 


r 


• i 

1 


• 




" J 










* 




\ 

1 

i 
» 


A- 

4) 




• 


- 


- 






i 
i 

j 

> 
i 


5) 




• 


! 




\ 




6) 


• 


- 




i . 








i 

i 


7) 








* * 






8) 










* 




9) 












* » 


10) 






c 








11) 














12) 










f 





17. *r. »ny of th.s. job. "cm jobs'? * 
(If y««) • Which on..? (Ch.ck column 16) 

If, umbras tS TIME BOtD VK.M* . go to qu.itlon 21. p. 5. S.ction 7. 

MW un^C >T TT W f- BTOtD TROCTAH r 5= to quMtion 18, p. 3, S.ction IV. 

«« M »»« ttxVTOC CCTA , 90 so qtaicien 63. P- 8. S.etion X. ^ _ 

-4- « 



J 
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section rv: 'soN-iaPLOYHSST ?mcr to the nssT jcb attsr cita 
• • (For Respcndencs Kho* w *rt «Cg aoaiCIXG Ac Tise Ended ?rogr*n> 



*You said that b4tv«en Inviag C2SA and , 



vou did 



r.oc work. X would Ufc* » ilk i few questions abcur rnic :i=»". 
13. Old you look for work during that.t^ae? 1 » Yes 2 » No 

19. How tuny weeks did you look for work during ^that tiaMl 

20. (If less cn*a 100% of the ti«e) during the cine when you did nor 
look fir work vbac were the reasons? (two responses) 



QX • Tield Overfilled r 

02 » Inadequate graining/Skills t 

03 • tJnsure of Hyself 

04 » Personal Probfeaa 

05 Child Car* 
^ 06 • Health 

07 • Pregnancy 



13 J. 



n 



19 

/ 

20, 



08 • Tired of Looking 

09 * Did Not Wane To Work 

10 • Collecting Sneaployoent 

Insurance 

11 ■ In CETA 
U • In School 

13 Waiting for Job To 3egin 

14 Mother 



LJJ 
Llj 



SZCTION V; OTOHMArrCN CN THE SIXST (1st) JOB A3TS5JI CSTA 

(for Mspondents who Held at least owe job After CSTA) v . 

"Stow I would like to get sore iniornation about the joh you held (are 

holding) at .. * _ " 

(ccooany'naioe) - <3oh title) 

1 21. How did you find out about thia 30b? _ 



3SL 



21. 



01 * Agency referral, 
7)2 • DCS 
' 03 • ?rivate Agency + 
04 - rriend/ftalative 

\ 

22. What does this company do?_ 



05 - Newspaper 

06 • Walk- in 

07 • School 



23. What was your starting hourly wage rate? 



-did 



24. Kow aany hours did you usually work per week? 

-i - Not Applicable *-3 ■ Mspondant Does Hot Know 



FOB. OASSBCCM SKILLS TRAINING AND 0JT ONLY: 

25. Has thia position' s work related to your CSTA training? I • Yes 2 • No 

26. Do you think you could hare porforaed the job's duties without the CSTA . 
training? • 

1 • Yea 2 ■ No 



27. Did your wage change on thia, job? 



28. What wes/is your jfinal/currcnt hourly wage race? 

(If no change^ record wage reported in Question 23) 



28. 



ja- 



zz. While working for 



(company nasw) 
1 • Yes 2 * No 



did your job* s duties change? 



29 



•u 



\ 



<•? YXS TO 29) 30. Old you have 1 new job <title ? SOC: 



30, 



-1 • Wot Applicable ; -3 • Respondent Does Not Know 



5 - 
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3$. Was Via aaw job a sraaotisn? 

1 • Yt«» : » Me 

. > > ^_ 

tck csr aso 0J7 SaaiSi 

*32- Wat chia position' » '-ork ralactd your 
CETA training? 



53 

3i. r . 



.59 
32. ! i 



1 • YtS 



2 * Ho 



■ 33- 30 you tbtcfc you could hava parferrad this 
job's dutits without tha OKA training? 



50 



1 « YaS 



No 



34. 



35. 



10 *36. 



Ail tainga considarad, how would you ract this asployar? 

1 » txtallant' 2 *» Good 3 - 7air 4 - Poor 

All thiaga conaodarad, how would you rata^this typa ot work? 

/l » Escallant 2'» Good 3 - rair 4 - Poor 
Why did you laava this lob? : — — 



\ mm in I 

35. □ 



36. 



a3 54 



SECTION 'Tlx axiJOKDESrS SE&ND (lad) JOB AFTSS. CETA. 

(TOT Baspoodanr,* Who Hava Hi Id AT LEAST THMS (31 JOBS A ftar CETA) 

»I would Alto lika to jat a littia t»rf\lnfor3*tion about tha job you hald at ' 
a* a ' - 



' 37, what vis your final hourly vaga? 



38* How many hours par waaJc did you usually work? f 

-1 » Hot Ap # pU=abla -3 - Mspondant Coas Mot Know 

^^39. Why did you laava this job?, , — — — 



"■'LULU 

69 70 

»- m 

J 71 72 



SECTION VII t SESPCNDENTii THI2D <3rd> JOB AFTER CETA* < 
SECTION VXIt ^ MfjyjmUntt who Hava Hald AT 1EAST WOE (4) JOBS ■Aftar CETA) 



"Tfcasa cjuastions rafar tt> tha job you hald at 



40. What was your, final hourly waga? 


40. S ' 


m 


t 




n 




41. Kow tuny hours par waaic did you usually work? 








•1 • Not Aoplicabla -3 Aaspondsnt Ooas Not Know 






Card Siaabar: 


* 




U 








3 


I. If. maabar: 




i.i 1 1 






9 


10 



^J^* 2 ' Why did you laava this job?_ 



42. 



SECTION Villi RESPONDENT 1 * (4th) JSTA ^ 

* (for Haspondant 1 s who Bava Hald AT tEAST riVE (3) JOBS, Attar csovj 



"Thaat quastion*^ rafar to tha job yon hald at 



4) 



- 6 
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;3. what wu your final hourly wage -race? 



44. How oany hours per week did you usually work? 

-1 » Slot Applicable -3 • aasponder.t Dots Hoc Know 



«. 1 ! 



^LO^'45. Why did you leave this job?_ 



J.L 



■ 1 



SZCriON IX; MSTOHDOT'S CnWCTT/LAST JOB 

(Tor **«?ondents Who neve H«ld A? I£AST TOO (2) JOSS After CTTA) 

-Stow I would like to gtt s»re inforaetion about the job you're holding row (last 
30b you held) at. ^- _ *• * ■ ■ — -* 



46. How did you find out about this job? 

01 ? Agency referral 
( 02 » 0£5 

, 03 • Frivmte Agency 
04 • rriendfrlelaiiive 



19 20L 



05 m Newspaper/*' 
OS • Walk-in 
07 • School 



46. 



4? ♦ What dots this company do?_ 



4t. What was your starting hourly wage rata? 



49. How many hour* do you usually work par week? ^ 

-1 » Not Applicable -3 «. Respondeat Ooas Not *aow 



v r03t CLASSROOM SKUXS^tHAEil^ AND OJT TTRUNTIS; 

50. Was this position 1 s work ralatad to your C5TA training? 

* •> . A ■ Yes 2 ■ Mo 

51. Do you chink you could have perfomed the job's duties without the 
CTTA tracing? * 1 » v:es 2 - No 




28 



•5,n 



52* Did your wage change on this job? 



53. Jlhat was/is your final/current hourly wage rare? 

(If no change, record wage reported in Question 49) ^/ 



53.$ 



54. While working for 

' 1 - tres 

(XT YES TO 54} 55. Did you have a new job title? SOCi 



did your job' s duties change? 
2 * Wo 



55. 









34 


^ 33 

*-lJ 

r 22 - 






1 1 



- 1 « Kot Applicable 
\ - 3 » fcespohdant Does Kot Know 



56. was/is the new Job a prcaaotion? 

1 • Yee 2 - No 



38 



FOX CSt AW) OJT TZXNXNZSS WHO CKANGtD JOBS CMX.Y: 

57. Wae this positions work related to your CTTA training? 

1 • Yes 2 • Ko * 

/' - 7 - 



198 



59. 



58. So you thin* you cculi nava ptrforatd sat ;ob's dutitt 
without cat SETA training? 1 • Vta 2 • So » 



thing* considtrtd. how would you ratt this tcployar? 



1 • Ixcailtnt 2 • Good 3 « rair 4 • Poor 

4 

60. Ail things considtrtd, how wold you ratt this typt oi work? 
1 ■ Exctiitr.t ^O-v^ood 3 • Fair 4 • Poor 

61. Is this Job bttttr thin tht first afttr CSTA? 
»1 ■ Yts 2 » No 

10 )62. Why did you ltava this job? , — - 



53. j I 



41 

60.4 j 

61. j 

.— ■ i f 

44 45 



62. 



sscticn x: roR respondents who are crar.rrLV -i^plcysd ' 

(!• Raspoadtnt it Currently gpploytd thit taction Z* Mot Aopllcablt , Prccatd 
to Stction XIX. f ?agt 9, Qutttion *0> . 



"You said you hart not worktd sinca_ 
you t ff tw 'quastiotis About this 



X would iikt to atk 



63. Uava you lookad for work? 



_ vatk ptriod." 
2 - So 



64. How autny of tht 
work? 



1 • Vat 

• waaka h*vt you actually looktd f 02 



€5. Art you cuxrtatly looking for a Job? 
(X? YSS TO 65) 

£6. Kow art you looking for t job? (two rttponttt) 



01 - Training Agtncy 

02 - ors 

03 • Privatt Agtncy 

04 • Friandt/Rtlativaa 



05 - stwapactr 

06 ■ Walk- In 

07 » School 



67. what do you think art tht rtaaons you art not ablt 
to find work? (two rttponstsi 



01 » No job 1 availablt 

02 • Diploytrt think too 

young/too old 

04 - Lack of axptrianct 

05 • Lack of tducation 

06 - Lack of raftrtnctt 

07 - Lack of tools* 

llcansts, ttc* 



(XT JJO TO 65) 



OS - Unguagt barritr 

09 ■ polict rtcord 

10 "*Lacktd transportation 

11 • Racial discrimination 

12 • Stomal discrinination 

13 - Othar rtascn (sptcify) 

14 - Don't know 

15 " Waiting for tchool/job 

to btgin 



Why art you not looking for work? (two rtaponsas) 



r 



% 0i ■ Maid Ovarf ilitd 

02 ■ 'inadaguafca fraining/Skillt 

03 - Onsurt of Jtysalf 

04 - ftrscnal Problatw 

05 - Child Cart 
OS - Ktalth 

% 07 » Jrtgnaficy 

08 » Tirtd of Looking 

69. Would you likt a job right now? 
1 - t M 2 - 



09 ■ Old Sot Want 

to Work 

10 - Co 11 acting" 

tfotftpioysttnt 
Xnsuranct 

11 ■ xn'crrA 

12 • In School 

13 ■ Xncarctrattd 



47 48 


*H : 1 




/ 




49 50 


66. 


j 




51 52 , 




; 1 




53 54 


67. 


! ! 


1 


55 56 




I 1 1 



57'*S8 



66. 



1 1 i 


\ 

59 


60 


1 


■ I 



/ JL. 

-■U 
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; 1 



\ 



7 



sEcrxca :<i> Rcsrcraatrs other ?csT-rRa;?Aa ix?r^:crs 
" ; (tck.aU. sespojcekts) 

^*"0. Sine* Uaving s!m CTTA progran on j**v« "/o u 

• attendad »chcol, * training program or tnintad m tiw aiiitary? 



». School 

draining Progran 
s. Military , 



J* 



I • Yas 

,2 • 110 vll Nto Ail. , # 
So To Saction 
?ag« 10) 

..' / * 

71. Art you currently atwndlag ■ Uetiwtitt narail ? 
1 - *as • 2 • No > 

A. ATHMDAMCS*!!! SCSOPL/TRAJPTIKG PSCCTAM 

72. what tyy« of school/training prograa *r« (vir«) you ittandlng? 

01 » 4-Yaar Coilaga 05 « ?rivat« Vocational School 

02 • Junior/Ccwunity Collet 06 « tanguag* iZSZ) 

03 »,Caaaral Acadmle High School 

04 • Public Vocational Technical 
High School 1 ) 

73. Art (vm) you atxanding full or part-tis«? 
1 • Full "* 2 * Part y ^ ' 

11 ) 74. What xrt you itudying? _ 



© 



75. bo W oany «***« did (hava) Vou att«nd(«d) tin school /training progran7 

i 

# 

(: y jjot CURRElTiV ATTENDING) 76. Did you.cotplttt tht progr«a7 



/ 



1 « Y«s 



2 " Ko 



(U SOT COKMHIWf ATTEHOIXC » 
AND H.S. INDICATED IS 72). 77. Did you r,«c«ira your diplcaa? 



1 « *•« 



Card Muabart 
I.D. thcabar: 

3, E2tt*XSTZD IN WE HIUTAJtf ^ 

78. ' Why did you join tin aiiitary ? 

1 * To rtcaiva training 

2 • Cartar Choice 

79. Cn vhat data did you join tha ailitary7 



2 ■ So 



3'. 



0 

□ 

I l 

65 

u 

67 

□ 



6a .59 



72. * ! ! 



70 > 



73.1 



7i. 



73 74 



3 * Ho battar opportuniti«» 

4 « Othar 



10 



79. 



li\ 80. What do you top* to laam (or *r« you 1 timing) In tht aiiitary? 



-eg 



1 2 



' 0 I * 





\ 

\ 









78. 



15 



"H — 3T 



80. 



r 



ERIC 
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- 9 - 



srcnojj zzz: sdzsxw or pwmrccsah iabor market ix?raicjcr 
?*xt A. Z*bo* force >erslct?aclon r 

(?ir. A only, applies to those rA«pondents who worked 1ZSS THAN 
52 weAkA' In thA One YeAr^erlod Prior to Application) 

-I need to ask you ioqi c^AAtions about the txpAriences ycu had before 

starting your CTTA propria cn ■ . According to the 

info nation wa received from * 

the following jobs during the yaar baiorA CSTA 

At 



. CTTA. */ou workAd ac ' 



11. 



frcn 



(30b titiA) 



At 

At ' 



for A totAl of 



(business) 



WAAkS worked. 



frca_ 
frcn 



to 

to 



(IX *ork history inforoAtion covats lass than 52* week*) 

Did you work during cat retaining _ w«*ks 
between „ And 7 v 



(IT TZS TO 81) 82. ApproxinAtAiy how many o'f those < 
did you work? 



VAtkS 



(IT 30 TO 81, CodA 32 rero) 

Calculate the totAi number of w«eka worked And proceed. 



33 tli total nuabAT'Of weeks worked it less than 52) 

During thA " w*eJu that ycu ware not working in the 

year before you entared CTTA did you look for a job7 



1 - Yes 



• S50 



[VT TtS TO 83) 84. Approxiaataly bow sany of those 



week* did you spend looking fcr a 300? 
(If none, codA 84 xtro) 

35. What do you think were the reasons you 

vera not able to find vork? (two responses) 



01 •• Mo jobs ivaiUblt 

02 • Employers think too 

young/too old 

03 - Uck of skill 

04 » Lack of experience 

05 - Lack of Aducation 

06 ■ Lack of r«f ATAneAS 

07 - Lack of tools. 

liCAniAS, AtC. 



08 
09 



11 

12 
13 



18 1) 



82. 



20 

83 *Lj 



22, 



23 



2'4 



85. 1 



Language barrier 
lictnsAS. ate. 
PoIIca raeord 
10 ■ Lacked transportation 
Racial discriainetion 
Sexual discrimination 
Oth ax reason (spacify) 
14 ■ Cou't know 



25 26 



(X£ number weak* workad And nvsabAT w*Ak* lookAd is lass than 52) 

88. What vara tha raasons you did not look for wcikl 

». (two rASpOftSAS) 

01 • field ovarf Ulad 

02 - Training inadequate 

03 • UnsurA of jsyself 

04 - Parscnal problems 

05 - Child Car* 

06 - Health 



27 28 



07 


• fregnancy 




30 


08 


m 7ixed of looking 






09 


■ Did not want to work 






10 


- collecting U.I. 




V 



IX r m CESK 

12 In school 

13 ■ Incarcerated 

14 «• Waiting for CETA/school/job 

to begin 



10 



■201 



V 



Part 3 Loncaat Jca £?«;f HUd ?r-sr ?~ Sr.^irir.c TTTA 
•:tow I would Ilka you to civuix About /our ant'ira vorJc aiitory* 
* 3*. Of all tte fuii-tipa 2«fct you hava *var haid. -fnac was your :ob 



titia (or dutias 



i-tisa jo)p* you : 
as) on tha :o*b h 



^ild for til* i.cn<;tsc period of tl^a? 37.. 

SCC: 



141 



•1 • »iot Ap?\icabla -3 - aaapondanc Soaa Hot Know 
39. Approxiaataly how long did you hold thia *3ob? (racord in vaafca) 



3S 



39, Approxiaataly what waa tUa hichast hourly wsga you aamad At this 38 
90. wh«n did you laava this* job? 



JLL 



90. 



SECTION XIII i 0CO33WP5IC AND ECCNOKXC CHAJACTSaXSTICJ U73ATS 

*B«fora andinq 
of inform* tier 

51.' Ara youj 



*Bsfora ending this intarviav X naad to updata a faw iaporxant oiacas 
of information." . 



1 • :urritd « 

2 • Singit (ntver uixri«d) 

3 • Widovtd 

* 4 • Sap*ratad or Divorcad? 

92* Bow oany dapandants do you hava? (Excluding Salf ) 

• (XT ONE OA MOKE DEPENDENTS) 93. Cf thesa dtptnd*acs. how 

4 aany at* und*r six yaars of ao;a? 

94. Art you providing aorf^than half of your family'* aarnad inecca7 



92.] I " 



93. 



53 



1 « Ya* 



2 - Ho 



' 95. Sine* laaving CTtA hava you Tacaivad any of tha following foos o. 
Public AsaistAnca raynants or Snaaployoant Ccco«n*Ationt (.thraa 
rasyonsaai if *ra*pond*t raeaivad no paynant. cod* M8") 



54 55 





01 ■ AFOC 

02 - rood Stacp* 

03 n D.I. 

04 * Ganaral Mliaf 


' 05 * Vatarans 3«n«f its 
06 - hadicald 
0? ■ SSX 

08 -'Social Sacuricy 

09 • wm/Anx: 




(XT YES TO 95) 
• 

i 


Ar« vou currant! y raeaiving (apaeif;/ 
tha typ«f indicAtad by ratpondant 
In quaation 95)7 

1 * Yaa 2 « No 


* • 


roft those wrra no h.s. oxtlomx. 


(SEE Q0ESTXCM 13 0M INTAKE) 




9?* Eava you rac«ivad * CED? 


1 « Yaa 2 » No 




i 

(IT TES TO 9?) 


90* Did you racaiva your CED vhila in 
CXTA or ainca laaving? 






1 - In CSTA 

2 - Sinea Laaving \ 



© 



96. □ 
61 ■ 

c 

98. p 1 " . 
65 6? 



99. What town cr city ara you living in now7 



9,9. 



EJffi OF INTERVIEW 



J 



Aftar taa iatamsw u rcsplttsd, toda ta« ffollovinc quastions £r=a 
locution racardad on zht suasary taolsi 



Card JJuabar* 



A. Muab«r o* vaa*a barwaan cats sndsd prpgrsa and ir.tarvisv: 

/ 

5. Cn chs data of taraiaacion did ths rsspondsnt havs t job? 



1 - Yts. 



2 - MO 



C. (I* Job ?Ucad) X* ths first: 30b difiartnt iron that indicated 
ac placaoaot? 



1 - Yss 



1 • \lo 



How eany w«a)cs did taa raspondant spand on taca 30b? 



i_L 



s. L 



cU 



1. first 3 obi 

2. Sacond#jobi 

3. Third, jobi 

4. fourth jobi 

5. Currant/last jobi 



2. 



4. 



5. 



13 14_ 

CO 



r 1 7 14 

m 

m 



eric 



E. Bow aany woaXs did taa raspondant spand not wording and not in 
school? 

1. latvaaa ESQ ot prograa and ri»i5 jobi * 

2. lafors StCOS© jobi 

3. Bafora TSXXO jobi 

t 4. *afora rCGKTH jobi 

# * » 

6. **tar CWKCrr/IAST Jcbi 

I 

I. tov »a»y vaaks did raspondant spand in school or a training 

program whan not working? * 
C, What was tha total macbar ot jobs raspondant hsld ainct itsvin* 

GET&? 

- 12 - 



2 
3. 

5, 
6. 



23 


24 . 


U 








u 


























I 




u 


U 








37 


to # 



•CD 



203 



•SOTS: Ci^/th* Follovint MUtlnr r*Xut Codas Or. SIC me SOC 3uastions. 



-1 » Not AppLicibl* 



-3 • Raspondane Doti.' % :cc Mnou 



^Information on TIXST ;ob: 




Ir.dustr*-* - SIC. 




t 




* 


! 
i 




















• 2. 


Occupation - SOC: 




« 1 


' I 


















• 


J- 


Data 5*t*n * 








'i 








H - 




» 










Daca rndtd 


1 i 


* <• 

i 




*'! 












53 






Infornation on SECOND Job: 

i * 


%• 1. 


Industry - SIC: 






i " . 


i 




















2 - 


Occupation - SOC: 


i 

r 




I 


.1 
i 



Card Nuabar: 
I.D. Muobar: 



TT 



*3> 


3. 


Data Satan: 


i 


1 

i 






-> 






15 


* 








20 




u , 


Data Sadad 


1 


1 




- 1 • 






















23 


J. Information on THISD Job: 


♦ i. 


Industry - 


SIC: 


























24 






27 


• 


• 2. 


Occupation 


- SOC: 




■_ 


.1 


i 












?t 














3. 


Data Sajan 








1 






■ i 




V 




















4. 


Data Sndad 




1 




t 


1 




1 




















42 


K. Information on FOURTH Job: 


•1. 


Industry - 


SIC: 










i 
























•2. 


Occupation 


- sec 






I 

! . 






i i 


• # 






















3. 


Data Sagan 


♦ 






T 


,1 
















4. 


Data Endad 










• 




• i 


















61 


Information on CtntMXT/LAST Job: 


•1. 


industry 


SIC: 













• 2. Occupation SOC: 



3. Data Jkfan 

4. Daca End ad 



45 



* 71 



K. Vara any*Jobs idantifisd as ?SE Jobs? - - - 

l - First 2 - Sacond 3'- Third 4 - Fourth 3 • Currant/Last ^- Ho M% j 

S. Is. tha raspondant currantly caployad? - 
2 • So - 



7.7 



7S 



1 - Yas 



2tT4 



csta t:tis 
< ?xRrrci?fcrr information form 

For Use In Conjunction With The- 
Participant Follow-yp Questionnaire 



Every box should be .'Hied in. If * question is not answered do not leeve cht boxes blank. 
Xsstaad, fill thaa in with ont of the following codes unless different Hissing value codes 
o are specified : 



8 - If the question is not applicable 

9 - If the Information is missing 



Terminee's Namsi_ 
Address: 



Telephone 



Other ?hone/Cont*ct Information 



Card Humben 

1.0. Number: 0 
|. Date of Termination! ■ 

2. Date of Snrollmenti » 

3. Date of Application! 

e> 

4. Prime Sponsor: t 

1 » 

2 « 

3 • 

5. The Si tie II Subpart of Termination: 

1 m Title XIB 2 » Title XIC 



23 



3 » Title 110 



SECTION Ii DEMOGRAPHIC INFORMATION 
e.-^Sex: 1 • Male 2 - Fenelc 



1 2 



16 



Ed 



22 



nn 



29 



30 

□ 



31 

□ 









32 


37 


\ 


7, Birth Date: 


7 




I 


1 1 1 



9 

ERIC 



2U6 



/ 



3. Race/ Ethnicity : ■ 

01 - Slacfc (non-Kispanic) 34 • Amari'tan Indian/AIeskan Native 
0: - Hhite (non-Hispanic) 05 - Anan/Pecific Islander 
03 - Hispanic • 

9. -'Limited English Speeding Abilif/i i • Ye*' 2 • Uo 

iC. Offenders X » Ves 2 • No 

:i. Handicapped: 1 • Ye» 2 • No 1 

12. Years of Education Cccpletidi „ 

13. High School Status* 1 » School Oropout/tfo GEO 

2 ■ High School Student 
* » 3 ■ High School Graduate/GED Recipient 

\i 4 ■ Post-High School Attendee 



14. family Status: 



1 ■ Singlt Parent 

2 * Parent/ 2-?arent family 

3 ■ Fatally Member 

4 • Non-Dependen* Individual 



15. Somber of Dependents Excluding T trainee ; (if nor.e record sere) 

16. .Military Status: i • Veteran 

2 » Vietnam Era Veteran 

3 • Special Disabled Veteran 

4 * Othtr 

# $ ■ :;ont iNever Enlisted/Served) 

17. .Ty?a{s) of Trensfer Payments Received: 

(List up to three) , 

1 ■ ATDC 3 » General Relief 5 - S.S.I. 

2 « a. I. 4 ■ Veteram Benefits 



r r ««* t*™*A raailv Income: S 


for 




(income) 


(* 


months) 


firaee Earned Personal Income: S 


for 




(income) 


(* 


months) 


Tn *..i nrh.r r Included) IncooeiS 


for 




~— — (income) 


(♦ 


months) 



36 39 



.□ 

41 



12 I M 



13 



14 



□ 

46 

□ 



4? 48 
^ 

50 

□ 



17 



17 



"ft 



53 



»5. 





5 




1 






i 






53 




62 


19 


5 






i 
• 








63 




67 


20 


5 


I 

1 


1- 









21. Source, Other (Included) Income: (List up to two) 

1 - Alimony 4 - Private Disability 7 • Armed Torcas 

2 - Child Support 5 - Rentals Retirement 

3 m Retirement Benefits 6 • OASI 



.□ 

69 

-□ 



ERIC 



21)7 



SECTICN :i: ?M-CS?A IA30R tfARJCET EXPERIENCE Sl'ttARY • SIX (c) MCtf 
PERIOD BEFORE CETA 



22, Tocai'suritr of WeeJcs Not Eaploytd During 6-Month Ptricd Before 
CETA vOO-26) : 



23. total N'usb*r of Weejcs Employed (All Jobs Added Together) During 
5-Month Period Btfore CETA' (00-26) s 



;Ti*.e ansvtrs to C^iestions 22 and 23 should sun to 26 weeks ) . 
24. Number of Jobs Held During *6-Month Period Before CETA: 



22 1 j 



23. I 



24 . 



Card Number: 
I.Q. Humber* 

SECTION PRE-CSTA WORK HISTORY 
A. :tost Recent Job Held Btfort CETA 
2£. Conpahy Maae: , . 



V 



HE 



SIC: 



25 



9 : 

rn 



Company Address:^ 



Kind of Business: 



000 • tfot Applicable/Information Missing 



12 



26. Job Title: 



DOT: 



.000.000-006 -(Not Applictble/Ir.formatidn Missing 

27. Final Hourly Wage: 

28. Hours Worked Per Wtek: 

29. Datt Began.: 

30. Date Ended: 

31. Number of Weeks Worked <pn Most Recent Job: 
B. Second Most Recent Job Htld Bsfore CETA 



26 L 



32. Company Ham*:_ 



Company Address :_ 



Rind of Business: 



000 - Not Applicable/Information Missing 



20 

van 



21 



24 



27 S 



25 26 



28 ; 



m 

32 



29 



30 



33 



38 



39 41 



31 L 



42 44 



SZC: 



32 



208 ! 



45 



33. Joo Title: , . 

000.000-000 ■ Not Applicable/Zr.foraation Hxssir.g 

34. Final Hourly wage: 

33. Rovers >forKed ?sr Wetk: 

36. Data Ssgan: 

37. Cata Ended: 

33. B uab *r of W*eks Worksd on Second Most R*c«nt Job: 



XT: 



33 



53 



i J j 



50 



36 



66 



S3 59 

.CD 

65 

rm 



37 ' 



— I 



38 



Cerd Humbert * 
1.0. Number: 

SECTION IV: PARTICIPANT'S C2TA PROGRAM HISTORY 

This soctlon gechers information About the tsrainse's CETA program se*usnce, starting 
with the Progrea of Tt mint t ion end working beck to the second end third most rscsnt 
programs* Parts A* B end C below atttmpt to identify seratntial, iiaultanaous or 
ovarlepping prcgrem components. <• 

SUfore beginning ^his itfction, it will be helpftil to identify end r.ote tht terminse'a 
three most recent progrens end the sequence in which tht services were provided. 

4 v ^ 

39, Totel Kuaber o£ CETA Progrens Under Titlss II* IV, VI end.VXXi 

A . program of Terninetion (Most Recent Program) 



39 



9 

□ 



40. Title 71 Siibpart: 

1 - Titlt IIB 2 - Title IIC 

41. Type of Progrea: 



3 » Title I ID 



□ 

11 12, 



01 - OJT 

02 - Work Experience 

03 - Skills Treining 

04 • Direct Rlecemant/Services Only 09 - PSE Job 

05 « Jo£ Search Assistance 10 - Other — 



06 • English es e Second Languegs 

07 ■ GEO 

08 - Basic Education 











13 


14 


42. 


i^iyy f«?*rvir* Provider) 






« 




1 








15 * 




20 


43. 


Date Begem 


43 






1 


1 








21 


16 


44. 


' Date Ended: 


44 


i 


i 







2U9 



45. tfumcar of Wttks: 



46. Training Occupation (If^ Skills. Training or CJT) 
000 • Not Appiicabls/ Intrusion Miss in? 



47. Industry Mi OJT). 



SIC: 



43 



(,0'0 ■ Not Applicabla»/Inf07aation Mining 
49. Final Hourly Wags (If OJT Or PSE Job) s 

49. Hour* Workad ?«r Wsak (If OJT Or PSE Joo) : 

50. Type of Agency (If PSE Job) : 

1 ■ City/County Government 3 ■ Othtr Govsrnasntal 

2 * Stata Government 4 • Community Sassd Organiaation/Non-?rofi 



43 

51. Job Titla (If PSE Job) i_ yyI>: 



«2_ 

000.000-000 " Not Applicable/Information Missing 

52. Rsascn for Participant's Tsrmination Froo^Prograa: 

01 - EntsrsX Employment. Sslf 08 - Was Enrollsd in Upgrading Skills 

Plscsd > ' Program and Sept Job 

02 - Entsrsd Employment, ^gtncy 09 ■ Tfmt Licit Expirtd 

Placsd 10 - Excstdtd Maximum Wags Rats 

03 - EnroUsd in Full -Tint 11 • Personal/Economic Problems 

Acadsaic or Vocational 12 • Transportation Problems 

School 13 •.Prsgnaney/Child Cars Problscis 

04 « Entersd Armad Forcss 14 « Oissatiffiad with Program 

05 - Intsr-Titls CETA Transf sr 15" ■ Finifhsd Program But Did Not 
* 05 • Enrollasnt in, Non-CSTA Fundtd Entsr Employment 

Manpowar Prograa 16 ■ Othtr — 

07 • Cooplsttd yrograa Objsctivss, % « 
Not Entailing Employment 

» 

53. Was Terminss, At Any Time Whils Participating in this CSTA Prograa, 
Also Enrollsd in Anothtr CETA Program? 

1 m yts 2 • No 

8. Stcond Most'aactnt Program 

54. Titls Through Which Prcgraa is Fundsd* 

1 • Titls IIB 4 - Titla IV » 

2 • Title IIC 5 - Titls VI 

3 • Titla IIO 6 * Titls VII 



55. Type of Program* 

01 • OJT 06 English as a Sscond Languagt 

02 • Work Expsritnct 07 ■ CEO 

03 • Skills Training 08 • Basic Education 

04 ■ Oirsct Placement/ 09 • PSE Job 

Sarvicts Only 10 - Other^ 

05 ■ Job Starch *" 

Assistance 



210 



55. Agency (Service Provider) : . 

57. Date Segant 

53. Date Ended* 

l 

5$. Huaber of Weeks* 

V 

60. Training Occupation (If Skills Training or OJT) : 

000 ■ Not Appiieable/Inforaetion Missing ^ 



53 



DOT: 



61. Industry (If OJT}^ 



SIC 









} 


66 








71 










i 

i | 










74 




59 




: 1 








75 


77 




60 




! 1 








78 


30 




' 51 







" 000 - &,t Applicable /Information Missing 
Card Number: 



U 4 I 



I.D. 


Number: 








1 UN 












9 12 


c2. 


Final Hourly Wagei (If 3JT or ?SE Job) : 


* 




. . i in. 

13 14 


€3. 


Hours Worktd Per Weelc (If OJT or ?SE Job) ; 








•> 












15 


64. 


Type of Agency (If Job) * 









1 » City/County Gov«rament 3 ■ Other Governmental 

2 ■ State Government 4 - Community Sased/Non-Profit 



65. Job Title (If ?SE Job)*_ 



, COT: 

65 


16 




2A 






i 

1, 


I 


M 




\ 



COO. 000-000 » Not AppiicaWe/Xnformation Missing & 

6$. Reason for Program Separation* I } 

01 • Coupleted Program 05 - Transportation Problems 

02 ■ Inter-Title CETA Transfer 06 - Preotiancy/Child Ca$e Problems 

03 ■ Change of Program Status 07 - Disiatisf ltd with Program 

04 » Ptrsonal/T&notkic Problems 08 ■ Other 



25 26 



67. Was Terminee, At Any Tint While Participating in this CETA Program, 
"Also Enrolled in Another CETA Program?* 



27 



I » Yes 



si « Ho 



68. (ir YES) * Was Tenninee Also Enrplled in Program of Termination? 
(Recorded in Part A) 



68 



28 

□ 



1 - Yes 



2 » No 



211 



C. third .Most Recent Program 

69. Tici«.Throu<jft Which Program is ?unded 



29 



1 • Title I1B 

2 - tttis rrc 

3 - Title ::d 



4 » Title VI 

5 ■ Titi* :v 

6 ■ Title vi; 



70. Type of Program: 

01 • oJT 06 - English at * Second Language 

02 - Work Experience 07 - GtX> 

03 - Skills Training 08 « 3 *»i£ Education 
/ 04 ■ Direct Placement/ ^9^«-PSEJob 

Services Only^--^ 13 - Other 
05 ■ Job Stare 
Assistance 



30 31 

I 



32 33 



71.' Agency (Service Provider) :_ 



71 



>. 

72. * Date Began: 

73. Mtt Ended: 



74. jjuaber of Weeks: 



34 



29 



72 I 



40 



73 I, I 



46 



4r 



49 



51 



75. Training Occupation (If SkiUs Training or OJT) 
' 000 ■ Not Applicable/Information ttissing 



DOT: 



76. Industry (If OJT). 



SIC: 



000 ■ Not Applicable/Information Hissing 
^77. Final Hourly Wags (If OJT or PSE £ob) : 



78.* Hours Worked ?sr Week (If OJT or PSE Job): 



75 

IB 
55 



52 


54 


1 







2> I 



58 



79. Typs of Agency (If PSE Job): 

1 ■ City/County Governmental 03 - Othsr Governmental t 
' 2 - Stats Government 04 -iffiommunity Based Organication/Non-Prof it 



59 60 

m 

€1 



80. Job Title (If PSE Job) :_ 



DOT: 



62 


70 


1 




11 




N 







000.000-000 ■ Not Applicable/Information Missing 

\ 

31. Reason for Prograa Separation! 

01 - Completed' Prograa 05 - Transportation Problems ' * 

02 « Inter-Title CSTA Transfer • 06 - Pregnancy/Child Care Problems 

03 - Change of Program Status 07 - Dissatisfied with Program 

04 - Personal/Economic Problems.^ 08 - Other , 
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92. Was Ttrainsa» At Any Tia« WhiU Participating ir. tail CSTA Program, 
Also Enroll** in Anoehar CETA Prcgrta? 



32 L 



1 - Y*t 



2 ■ No 



7< 



93. ( TF : WssJTsrainsa Alio Enrolls^ in'ths S«csnd Mcst gtcjnt Program? 
~«c (Racordad in Pan 3) ) 



1, ■ Yts 



2 - Mo 



Card Nuabar: 



.0. Num&arr 



SECTION V : PLACEMENT INFORMATION 
34. Data B#<?*n Post-CETA Jobt 



95. rap any Na»*t_ 
Add£«sst * 



SIC: 



Kind of Businass: 



. 0C0 Hoc A^plicabla/Inforaation Mining 
Tt^aphona Jtanbart 

Contact Samai : 



18 



86. ' wob Titiar 



COTi 



86 



000.000-000 Not Applicabla/Information Missing 

87. Beginning Hourly Waga Raca» 

88. Hours of work Par Vaakr 

89. Encarad a School or Non-CSTA Training Prograa Aftar CETA? 

1 - ¥as. School 

2 ■ Vas, Non-CETA Prograa 

3 » No 



School or Non-CETA Prograa Na»a:_ 
Addrasst 



EU 



14 



84 



17 



35 



iU 



27 



87 S 



1C 



26 



30 



31 32 



88 



89 



23 

□ 



Talaphonai_ 



Contact Naaai 
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Sample Introductory Letters and Business Reply Postcards 
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Dear CEIA Participant: 

In an effort to find out whether your employment situation has improved as a 
result of your participation in the CETA program, we are conducting a follow-up 
study of the CETA program. The purpose of this letter is to ask for your help. 

Although participation in this study is voluntary on your part, we would 
appreciate your help in our effort. We will, be contacting you m about 2 to 
3 weeks for an interview" about your personal experience. The interview will 
toJ^acTduring the early part of the evening and will last about five minutes. 
This isyour chance to express your views (good or bad) about the program as it 
has affected you. Your contents will help future CETA participants* 

Enclosed you will find a pre-addressed postcard. If this letter has reached 
you at an address other than the one originally typed on the envelope or if your 
telephone number has changed since you entered the program, please complete and 
return the postcard. 

You cambe- assured that any information which you provide in the interview ' 
will be held in the strictest of confidence. Thank you very much for your coop- 
eration in this -study. 



Suggested Text of Letter for 
Individuals to be Interviewed by Telephone 
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Dear CETA Participants: 

' in an effort to find out whether your employment situations has proved as a 
result of ycur participation in the Cm program, we are conducting a follow-up 
study of the local CBm program. The purpose of this letter is to ask for your 
help. 

K Although participation in this study' is. voluntary c« your part, we vo^d , 
ai^eciatewihelp in our effort. ■ We will be contacting you by telephone in 
^ffto fveeS tolchedule a personal visit to ask you about your e^erience 

cm Warn ThTSterviw will take place when it is convenient for 
y^^^'IaS^outlive minutes. This is your chance to express your views 
(goo?or bad) about the program Wit has affected you. Your Garments will help 
future CEEA participants. 

, ' 'Enclosed you will find a pre-addressed postcard. If this letter has reached 
you ata^address other than the 'one originally typed on the envelope, or if your 
tSeptox? numbeThas changed since you entered the program, please complete and 
return the postcard. * 

\you can be. assured that any information which you provide in the ^terview 
wili be held in the strictest of confidence. Thank you very much for -your coop- 
eration in this study, 



Suggested Text of Letter for 
Individuals to be Interviewed in Person 
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BUSINESS REPLY MAIL 

rvtsreuss nmrr wo. ocooo ahttown. u.s.». 



AT LEAST 



PCCTAC* WW. M PNO rt AOOAtSSM 

-XYZ Sales Company 
Attn: Accounts Payable 
P.O. Box 1234 



Anytown, USA 00000 



Requireroeots fear Business Reply^ Postcard 
Front Side 
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Pleafe^gpomplete this card and put it in the mail. 



NAME 



(First) ' 



ADDRESS 



(Number) 



(City) 



Phone Number 



(Last) 



(StreetX 



(State) (Zip Code) 



Suggested Text for Reverse Side of 
Business Reply Postcard 
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* 



it 



Name: ^ 


Current Address: . , 




You may contact me at this telephone number: 




« 


Area Code: 






Best day(s) and time(s) to call me: 


i 




1) * in the | | morning 
Day(s) - , ' 


j | afternoon 


] | evening 


2) in the | | morning 
Day(s) * 


j | afternoon 


] 1 evening 











Suggested Text for Reverse Side of 
Business Reply Postcard 

/ . 

V * % 

V 

\ 




0 
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A REVIEW OF SAMPLING PROCEDURES 



Introduction 

The cost of collecting follow-up data is heavily influenced by the number of 
participants a prime sponsor includes in the study. One step which prime sponsors 
can take to address the issue of cost is to systematically sample participants. The 
purpose of sampling is to select a representative subset of participants whose post- 
program behavior, when analyzed, will generate the same statistical results as that 
of the entire population. By decreasing the number of participants used to draw 
statistically reliable conclusions, the overall cost of the follow-up study can be 
reduced. 

Since the drawing of statistically sound inferences requires a specified number 
of participants, the success of sampling as a cost-cutcing option is critically tied 
to the total number of participants a prime sponsor has enrolled, in its programs. It 
• is unlikely that small prime sponsors will have sufficient numbers of participants to # 
be able to realize cost savings through the use of sampling. -In the case of a small 
prime sponsor, the number of participants is initially very small and the accuracy of 
the evaluation results may be severely threatened if any sampling is undertaken. Al- 
ternatively, a large prime sponsor may be able to rigorously analyze the participant _ 
population through the examination of a properly selected sample of participants. 

This appendix treats several d^mat issues rrime sponsors will need to consider 
when developing^ sampling procedure. First, prime sponsors must determine the optimal 
size of the sample while considering any resource constraints they may face. Second, 
prime sponsors should consider the possible advantages of stratifying their sample in 
order to generate statistically sound results on particular subgroups or "strata" of 
interest in the population. Finally, prime sponsors will need to identify the_ specific 
procedures they will use for actually drawing the sample. Each of these issues is 

discussed in more detail below. \ ► 

I 
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Sizing the Sample 

• Prime sponsors which suspect they are large enough to realize a reduction in 

costs by means of campling must determine the optimal size of the sample. The optimal , 

size of this sample will depend primarily oh three considerations : the statistics to » 

/ ' 

be calculated using the sample, the percentage error which will be ^tolerated in the 

analyses, and any. resource constraints the prine sponsor- faces' which will limit the 

1 

total number of participants who can be surveyed. ' 
- Ideally, the statistics that will be calculated from the follow-up data, given 
a level of tolerated error, dictate the appropriate size of the sample of participants 
used. Suppose the prime sponsor wants to use the starting hourly wage -rate on the first 
job the participant held following the program as a measure of program success. The 
follow-up data which are collected will allow for the calculation of the average starting •. 
hourly wage rate of the sample group," call it X. The prime sponsor must determine how 
large the sample must be so th&t X very closely approximates the true average starting 
hourly-wage rate, u , in the entire'population of participants. To do this, statisticians 
use the Z statistic. The Z Statistic is calculated as: 

where c is the standard deviation of the starting hourly wage rate of the population, 
and n is the size of "the sample. The 1 statistic is used because it has certain 
convenient properties. In particular', it has a standard normal distribution, which 
means that the mean of 2 is zero and the standard deviation is one. Statisticians 
also know that if X - U , then there is a 95 percent probability that: 

* " 

-1.96 < 2 < 1.96 * 
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Therefore, the sample size necessary for there to be a 95 percent probability that 
X • U is represented as: 



n 



1.96 o 
X - u 



. ' To determine the appropriate ^rize of the sample, n, the standard deviation, a, 
must be known and the acceptable level of error in 3C identified. As the actual value 
>pf a is not known,, an astimate must be found which bears some relationship to the 
true o. One source of dataprime' sponsors have available for estimating the standard 
• deviation is the' Management Information System (MIS) . Prime sponsors are required to 
obtain an hourly wage rate for each participant placed in a job upon termination'. From 
this information, an estimate of the mean wage rate, u , and the standard deviation, a, 

can. be obtained. * 

While it is possible to calculate a sample size which minimizes the error, pn fact 
the determination of the optimal- sample .size involves a slightly more complicated set 
of considerations.' To illustrate, suppose the average hourly wage rate is estimated at 
$4.00 and that the standard deviation is estimated at $1.50. The required sample sizes 
for these estimates for different levels of tolerated errors are shown below. 

i • 

Percent of Error AllowaSle X - u - -2- 

5% . $.20 217 

2.5% \ 5.10 865 

1% - $.04 " 5,403 



^As can be seen; increasingly stringent levels of acceptable error require the use 
of a larger sample size and, therefore, increase the cost of the follow-up survey. 
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Those prime sponsors with tight resource constraints may need to accept higher levels 
of potential error in order to keep the cost of their participant follow-up as low as 
possible. In fact, in some cases, prime sponsors may find that there is a maximum 
number of participants they can afford .to sample given staff* and resource constraints. 
In these situations, while the sample size, n, may be known, it will be necessary to 
calculate the error implied by such a sample s **|§3 * 




Stratifying the Sample 

While\he purpose of the prime sponsor evaluation system is to study the overall 
functioning of Title II programs, prime. sponsors may find the performance of select , 
program activities or population subgroups to he of particular interest. In order to 
insure that sufficient observations of these subpopulations are generated to allow 
reliable statistical analyses to be 1 conducted, prime sponsors tpay need to use a 
"stratified random sample". This approach entails dividing the population into 
mutually exclusive categories of interest ("strata"), and drawing a random sample from 
within each of these. This procedure is particularly important in instances for which 
taking a random sample across tSe entire population may mean that 'the sample includes 
so few participants in nome subgroup that analyses of the subgroup will have high 
error levels associated with them. .By stratifying the sample arid sampling more than 
the proportional number of participlnts in the subgroups , the error in statistics x 
calculated for these subgroups canie reduced, usually with little effect on the 
currdr levels of the rest of the sample. 

For example, suppose a prime sponsor is interested in the difference between 
the average smarting hourly wage rates of AFDC recipients, constituting 12 percent 
of the population, and non-AFDC recipients, constituting 88 percent of the population. 
This prime sponsor can afford to sample '1500 participants. When a random sample is 
drawn across the entire population of participants, just 180 participants, or 12 per- 
cent of the sample, will be, AH)C recipients. . As can be seen in Table H-l below, after 
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Table H-li ' Comparison of a Jlandom "Sample and 
Two Stratified Ramdon Samples 



Random Sample, 
Percent of Sample 
Number of Participants 
Mean Watfe Rate 
Standard Deviation 
Error 

Stratification Plan ffl 
Percent of Sample 
Number of Participant^ 
Error 

Stratification Plan #2 
Percent of Sample 
Number of Participants 
Error 



AFDC Recipients 
► 

" ' 12 

180 * 
$3.68 
$0.89 
3.5% ($0.13) 

20 

* 4 

300 

' 2.7% ($0.10) 

40 
600 
1.9% ($0.07) 



^ Non-AFDC Recipients .* 

88 
1320 
$4.01 
$1.32 
1.8% ($0.07) 

80 
1200 

1.9% ($0.07) 

60 
900 

2.1% ($0.09) 



i 
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calculating th* mean wage rates and standard deviations for AFDC recipients and non- 
AFDC -rec'ipient9 , it is apparent that the error associated with the statistics for 
AFDC recipients is twice as large as the error for non-AFDC recipients. 

Through stratifying the sample and overrepresenting AFDC recipients, 'the error 
associated with the statistics calculated for AFDC recipients can be greatly reduced 
with little effect on the error of the statistics for non-AFDC recipients. For 
example. Table H-l presents the results <?f two stratif ication'plans.'^ In Rlan 1, 
AFDC recipients constitute 20 percent of the sample, and non-AFDC recipients 80 per- 
cent* As-can be seen from the table, even" this increase in the number of AFDC 
recipients from 180 to 300 has allowed a decrease in the error of the mean wage of 
AFDC recipients from 3.5 to 2.7 percent while only permitting an increase of .1 per- 
cent in the error associated with the mean wage rate of non-AFDC recipients. 

Stratification plan 2 increases the number of AFDC recipients to 600 ; or to AO 
percent. This allows the error associated with the AFDC recipients to fall. from 2.7 
percent, in plan 1, to 1.9 percent^ The error of the mean wage rate of non-AFDC 
recipients has only Increased .2 percent ' to 2.1 percent, still close to the original 
level of 1.8 percent. 

It is important to note that when using a stratified sample to calculata esti- 
mates of statistics on the entire population of participants, the sample must be 
weighted to reflect the true' proportions of the subgroups. In Che above case, -the 
stratified samples will need to be weighted to reflect the true proportions of 12 
percent AFDC recipients and 88 percent non-AFDC recipients. This means that all non- 
AFDC. recipients in the stratified sample will be given a weight greater than one, as 
they are underrepresented, and all AFDC recipients will be given a weight less than' 
one, as they are overrepresented. 

For example, in the second stratification plan shown on the Table, a ssnple was 
drawn with AO percent AFDC recipients and 60 percent non-AFDC recipients. To weight 
this sample for calculating estimates of statistics., for the true population, AFDC 
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recipients would be multiplied by a weightings factor of 12%/40%, or where/s 
non-AFDC recipients in the sample would be weighted by a factor of 88%/60%, or* 1.47. 
By weighting the sample in this manner, reliable estimates of statistics can be 
calculated using the stratified sample to represent the true population. 

One final consideration when determining the size of the sample is that prime 
sponsors should remain aware that. not all participants can be located and interviewed. 
Thus, for instance, if only a 50 percent interview completion rate is expected, the 
sample size may need .to be doubled to' insure, the same level of statistical reliability. 



j 
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Drawing the' Sample 

Ohce the prime .sponsor has determined the size of the sample, the next step is to 
actually draw the sample. The sampling procedure will vary depending upon whether the 
prime sponsor has an automated or manual MIS. For automated systems, the procedure for 
selecting a sample is generally also automated and can be performed merely by specifying 
a set of parameters for the^computer . * For manual systems, however, the procedure is 
somewhat more involved. Some suggestions for performing a manual sampling procedure are 
outlined, below. 

The procedure for manually selecting a sample is the same whether the sample is 
to be drawn from the entire population of participants or from participants in only one 
of many possible strata. If the sample is stratified, a sample for each strata is drawn ■ 
separately. Suppose n individuals are to be included in the sample. To identify the 
particular n individuals, n random numbers must be drawn from a random number table. 
Each of the n random numbers must have a value less than the total number of terminees. 
If 4 for example, there is" a total of 750 terminees in the strata or the entire population 
the random number table must be searched until n random numbers are^ found which are less 
than or equal to 750. Duplicate numbers should not be selected. 
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The n random numbers identify the positions in the prime sponsor file held 'by 
those individuals who are to be included in the sample. Prime sponsors generally 
maintain their participant records in some systematic fashion, although, since there 
may be differences among prime sponsors in the manner in which these files are kept, 
it is difficult to generalize. Frequently, however, these files are maintained in 
numerical order according to an identification number assigned to participants at the 
date of- enrollment. Thus, once the n random numbers are identified and the places 
•in the file held by terminees who are to be included in the sample have been identified 
the identification numbers, names and addresses should be/recorded. For instance, if 
there are 750 total terminees, one number which could J| drawn as a random number is 
176. In this case, the 176th individual in the prime sponsor 'file should be included 
in the sample. Through this process, a list of terminees who will .constitute the sampl 
should be constructed. 



r , 
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APPENDIX I 

A Review of Costs of Operating Local Prime Sponsor Follow-Up Systems 
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A REVIEW OF COSTS OF OPERATING LOCAL PRIME SPONSOR FOLLOW-UP SYSTEMS 
» 

t 

One of the -more important details of the design^and implementation of a ( 
local- follow-up system is the estimation of costs and the development of a budget. 
• As is the case with all activities a prime sponsor undertakes, the successful 
. operation of -a follow-up system requires sound financial planning and management. 
" Therefore, whether the work is performed in-house or by "an outside contractor, 
reasonably precise cost estimates are necessary. The purpose of this section is 
twofold. ■ First, the section presents a framework from within which prime sponsors 
can estimate costs". Second, the section presents an illustration of two typical 
budgets based upon projects which follow-up different numbers of terminees. It is 
hoped that this appendix will serve as a guide for prime sponsors attempting to 
' formulate or review proposed budgets. p 

A prime sponsor follow-up system will incur costs in four broad budget 
categories^ regardless of the exact operational or organizational approach adopted: 

v / 

• i. Professional Staff Salaries 

The first category includes wages and fringe benefits of staff associated 
with data analysis, report writing, the management of .data collection activities, 
and the cleaning, processing and programming of data. 

v * 

2, Data Collection and Processing Costs 
— • 

The second category includes wages and fringe benefits of data collection 
staff including interviewers, coders and keypunched. In addition, this category 
also includes the actuai computer costs of cloning, processing and programming 

' I * 

follow-up dat&. t * — A 

3. Other Direct/costs 
The third category includes any wag4s and- fringe benefits a^ociated with 

secretarial work as well as costs incurred for postage, travel,/ telephones, print- 
ing, copying, supplies and any other office services. 
A. Indirect Costs 

The final category includes any expenditures for overhead costs. 
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the .cct of operating a -follow-up system is difficult to estimate since 
the prim, sponsor can exercise considerable latitude in selecting the organiza T 
tional approach which meets local needs. Services may be procured through 
various' combinations of i.n-house and external resources depending upon tl» capa- 
bility and flexibility of prime sponsor staff and the availability of outside 
•- contractor,. For instance, a prime sponsor may decide to contract with an outside 
bidder to conduct follow-up interviews, and utilize in-house* staff to process and 
analyze the information. . Alternatively , a prime sponsor may determine that a 
cost efficient and effective ,ise of resources would be to utilize existing in- • 
house staff to conduct follow-up interviews as well as analyze the data. * 

The illustrative cost estimates presented in this appendix are based upon a 
follow-up system characterized by the following parameters: 

- The use of an outside contractor to secure the majority of 
administrative, technical and operational follow-up ser- 

* * 

vices. This includes the location, contact and interview j « 
\ of former participants, data analysis and report writing. / 

- A one year contract which collects data on a continuous { . 
• year-round basis. 

The use of a six month. follow-up period. 
' - A manual MIS which necessitates that participant contact » 
information and personal, programmatic and pre-program 
labor force data be generated using a manual transfer ^ 
procedure* 

- . The use of the telephone as the primary method of participant 
contact. 

The use of the survey instrument contained in Appendix E. 
An interview completion rate of ,50 percent. ' 
Table 1-1 presents estimates of annual budget figures for comprehensive follow- 
. up systems established in prime sponsors of two different sizes. Prime sponsor A 
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Table 1-1: Estimated Annual Coats for Comprehensive Follow- Up Systems 



Prime Sponso?*& Prime .Sponsor B 

(750 terminees) (2500 terminees) 



SALARIES 



. Manager of Data Collection Activities : 
Duties: insuring procurement of MIS data; 
hiring, organizing and training. interviewers; 
implementing location strategies; supervising 
all quality control activities; organizing 
mailings. * ' 

Data Analyst /Report Writer : 
'Duties; compiling and analyzing all follow-up 
data; producing one comprehensive follow-up 
report pv year; presenting report findings 
to relevant prime sponsor staff and advisory 
bodies. Should have background in CETA and/or 



labor market analysis. 



Fringe Benefits ; 

Includes 24 percent fringe benefit costs on 
Data Collection Manager and Data Analyst 
* salaries. w * 

^Data Processor (technician) : „ ' 
Duties: compiling a list of terminees with' 
contact information (names and addresses) ; 
getting dat'a onto the computer; cleaning the 
data; organizing the data for initial tables. 
Can often be a graduate student. 



DATA COLLECTION • 



$J,500 
(half time) 



$3,500 

(full 1 time during 
2 months of report 
writing) 



$2,640 



$4,772 . 

(full time during 
2 months of report 
writing; 6 N days/ 
month for rest of 
year) 

S 



Interviewing : * 

Includes collection of demographic information 
and pre-program labor force histories from MIS 
records, participant search time, interviews, * 
checking and\ coding of questionnaires. Calculated 
at 1 hour per. non-completed interview and lh hours 
per completed interview @ $4.50 per hour* 

Keypunching : . * 

Initial data file creation, assuming 50 percent 
contact, rate and including verification, @ 
$2 per terminee. * 

Computer Costs : 

Includes most substantive programming and data 
runs. 

5 • 



$4,220 



$1,-500 



$2^000 



$3.5,000 
(full time) 



$ 3,500 
(full time 

during ,2 months 
. of report 

writing) 



$ 4,440 



-'$ 6,158 r • - * 
(full time 

during '2 months* 

of -report 

writing, 9' days/^ 

month for rest 
- of year) 



$14,065 . ' 



$ 5,000 



• $ 3,000 £ " 
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Table 1-1: (continued) 

0 



Prime Sponsor A Prime Sponsor B 

"(750 terminees) (2500 terminees) 



■ OTHER-DIRECT COSTS 

Secret arial ( 



Includes typing of originals of al£ letters $1,000 $1,200 

and typing of follov-up xreports; other 
miscellaneous duties such as copying and 
mailing. 

Postage ? c Qnn 

Includes initial mailing plus follow-up $-270 ? 900 

letters for all terminees @ $.18. 
Copying and Supplies ; 

Includes copying of ^questionnaires, $ 350 ^ 

intrpductory and follow-up letters; 
. other supplies. 

'4 

Telephones : 
"'Varies considerably depending upon 
geographical dispersion of prime sponsor; 

no ♦estimate "given. * , ^ 

Travel : . , v 

Depending on the size and geographical 
location of the prime sponsor and thte 
. follow-up operations, travel may or may 
not be required; no estimate given. 

V 

INDIRECT COSTS 

All associated, overhead costs, estimated $ 3,330 $ 6>527 

at 12 percent of the total. 



TOTAL COST 
* COST PER TERMINEE 
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$31,082 $60,915 
$41 $ 24 
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includes 750 ctraitues in its follow-up study while prime sponsor B attempts to - 
follow-up 2,500 terminees. The amounts shown in the -table are only estimates. 
If outside contractors are used, -the bids submitted may vary from such estimates. 
It should be noted that, since the cost of living differs significantly among 
different regions of the country, prime sponsors may need to revise these esti- 
mates based upon local costs. 

As can be seen by comparing the budget estimates for the two prime sponsors, 
the costs of some line items remain almost the same for both budgets, whereas the , 
costs of other line items .increase' substantially from prime sponsor A to prime 
sponsor B. Costs which vary directly with the number of terminees are called 
variable costs, while those which stay fairly constant regardless of the number 
of terminees included in the follow-up study are referred to as fixed costs. For 
example, cosbs for such budget items as. interviewing and keypunching are "variable" 
costs in that they increase proportionally with the number of terminees. In both 
budgets, the keypunching of data costs are estimated at two dollars per terminee, 
so that the ratio of keypunching costs between prime sponsors A and B is the same 
' as their,ratio of terminees. However, the staff salaries of the data analyst /report 
writer and the data processor are about the same for prime sponsors A and B. 
Generally, such professional staff salaries are "fixed" costs, which means that as 
the number of terminees increases, the proportion of overall expenditures devoted 
to such costs tends to decrease. In the case of prime sponsor A, 60 percent of 
the overall expenditure is devoted to professional staff salaries, whereas only 48 
percent of the total expenditure of prime sponsor B is devoted to these salaries. 

As a result of the smaller proportion of prime sponsor B's budget devoted to 
fixed costs, the cost per terminee incurred by the larger prime sponsor ($24 per 
terminee), is substantially lower than that of the smaller sponsor ($41 per terminee) 
This suggests the desirability of identifying ways to organize local prime sponsor 
follow-up systems so that th'ey may benefit from^these economies of scale. For in- 
stance, if se-iral prime sponsors cooperate for the purpose of procuring follow-up 
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services, they may be able to share the cdst of one data collection supervisor, 
for the joint project ♦ Without a cooperative effort, each prime sponsor would in- 
cur the entire cost on its own. 

Co9t savings may also be realized if the prime sponsor can rely upon an auto- 
mated MIS. t Unlike the example in the budget above, the generation of contact in- 
formation, demographic, programmatic and pre-program labor market information need 
not rely upon the time- consuming manual transfer of data. The existence of an 
automated system allows the relevant data to be generated^ and organized in a very 
cost efficient manner through the simple creation of a separate data file to use 
in the follow-up process* 

The prime sponsor <?an also economize by relying on in-house resources where 
the additional expenditures of the prime sponsor would be less than the cost of 
similar services obtained from an outside contractor. For example, a prime sponsor 
may wish to rely upon 'in-house secretarial assistance, particularly in instances 
where there may be an underutilization af-^fiese" resources. Similarly, sponsors may 
find it more cost efficient to utilize their own computer facilities rather than * 
those procured by an outside contractor. 

> 

Finally, it should be noted that certain cost savings can potentially be 
realized through the negotiation process with an outside contractor. This is par- 
ticularly relevent when negotiating with universities and colleges which often 
have considerable latitude with respect to in-kind contributions and overhead costs. 

In vddition, educational institutions have demonstrated a capacity to utilize 

v ♦ 
flexible and creative forms 'of reimbursement which may affect total contract costs. 

I 

For instance, students used a^ interviewers have received academic credit for their 

work while faculty members hired to analyze follow-up data and write reports have, 

. ' v 
on .occasion, been pajf^ially compensated through reduced teaching schedules. Prime 

sponsors are encour&gifed to explore these options in the process of establishing a 
cost efficient local follow-up system, ^ 
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